[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:48:53 dist-test-slave-56vw krb5kdc[16877](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:48:53 dist-test-slave-56vw krb5kdc[16877](info): set up 2 sockets
Sep 28 22:48:53 dist-test-slave-56vw krb5kdc[16877](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 22:48:54.415772 16870 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Sep 28 22:48:54 dist-test-slave-56vw krb5kdc[16877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:48:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:48:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.121.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
I20260928 22:48:54.542780 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:42783
--webserver_interface=127.16.121.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:36543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:42783
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:48:54.625962 16890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:54.626199 16890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:54.626223 16890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:54.627430 16890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:48:54.627476 16890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:54.627493 16890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:54.627506 16890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:48:54.627519 16890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:48:54.629051 16890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:36543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:42783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:42783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.121.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:54.629318 16890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:54.629580 16890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:54.634790 16895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:54.634950 16890 server_base.cc:1034] running on GCE node
W20260928 22:48:54.634776 16896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:54.634773 16898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:54.635354 16890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:54.635679 16890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:54.636868 16890 hybrid_clock.cc:648] HybridClock initialized: now 1790635734636861 us; error 54 us; skew 500 ppm
Sep 28 22:48:54 dist-test-slave-56vw krb5kdc[16877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635734, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:48:54.639879 16890 init.cc:377] Logged in from keytab as kudu/127.16.121.190@KRBTEST.COM (short username kudu)
I20260928 22:48:54.640384 16890 webserver.cc:460] Webserver started at http://127.16.121.190:44579/ using document root <none> and password file <none>
I20260928 22:48:54.640651 16890 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:54.640703 16890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:54.640843 16890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:54.642201 16890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "88ee8fd3b6b044349626236bb83cdb62"
format_stamp: "Formatted at 2026-09-28 22:48:54 on dist-test-slave-56vw"
server_key: "be63ac850232584d190133e46ad92e4e"
server_key_iv: "b9cf85f03c4463f7fde343261d962b3d"
server_key_version: "encryptionkey@0"
I20260928 22:48:54.642617 16890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "88ee8fd3b6b044349626236bb83cdb62"
format_stamp: "Formatted at 2026-09-28 22:48:54 on dist-test-slave-56vw"
server_key: "be63ac850232584d190133e46ad92e4e"
server_key_iv: "b9cf85f03c4463f7fde343261d962b3d"
server_key_version: "encryptionkey@0"
I20260928 22:48:54.644155 16890 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:48:54.645077 16905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:54.645303 16890 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:48:54.645483 16890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "88ee8fd3b6b044349626236bb83cdb62"
format_stamp: "Formatted at 2026-09-28 22:48:54 on dist-test-slave-56vw"
server_key: "be63ac850232584d190133e46ad92e4e"
server_key_iv: "b9cf85f03c4463f7fde343261d962b3d"
server_key_version: "encryptionkey@0"
I20260928 22:48:54.645571 16890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:54.661516 16890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:54.664264 16890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:54.664455 16890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:54.669081 16890 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.190:42783
I20260928 22:48:54.669124 16957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.190:42783 every 8 connection(s)
I20260928 22:48:54.669536 16890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:48:54.670473 16958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:54.672847 16958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88ee8fd3b6b044349626236bb83cdb62: Bootstrap starting.
I20260928 22:48:54.673400 16958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88ee8fd3b6b044349626236bb83cdb62: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:54.673640 16958 log.cc:824] T 00000000000000000000000000000000 P 88ee8fd3b6b044349626236bb83cdb62: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:54.674448 16958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88ee8fd3b6b044349626236bb83cdb62: No bootstrap required, opened a new log
I20260928 22:48:54.676307 16958 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88ee8fd3b6b044349626236bb83cdb62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88ee8fd3b6b044349626236bb83cdb62" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 42783 } }
I20260928 22:48:54.676467 16958 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88ee8fd3b6b044349626236bb83cdb62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:54.676494 16958 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88ee8fd3b6b044349626236bb83cdb62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88ee8fd3b6b044349626236bb83cdb62, State: Initialized, Role: FOLLOWER
I20260928 22:48:54.676648 16958 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88ee8fd3b6b044349626236bb83cdb62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88ee8fd3b6b044349626236bb83cdb62" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 42783 } }
I20260928 22:48:54.676736 16958 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88ee8fd3b6b044349626236bb83cdb62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:48:54.676762 16958 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88ee8fd3b6b044349626236bb83cdb62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:48:54.676795 16958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88ee8fd3b6b044349626236bb83cdb62 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:54.677352 16958 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88ee8fd3b6b044349626236bb83cdb62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88ee8fd3b6b044349626236bb83cdb62" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 42783 } }
I20260928 22:48:54.677455 16958 leader_election.cc:302] T 00000000000000000000000000000000 P 88ee8fd3b6b044349626236bb83cdb62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88ee8fd3b6b044349626236bb83cdb62; no voters:
I20260928 22:48:54.677624 16958 leader_election.cc:288] T 00000000000000000000000000000000 P 88ee8fd3b6b044349626236bb83cdb62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:48:54.677655 16961 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88ee8fd3b6b044349626236bb83cdb62 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:48:54.677798 16961 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88ee8fd3b6b044349626236bb83cdb62 [term 1 LEADER]: Becoming Leader. State: Replica: 88ee8fd3b6b044349626236bb83cdb62, State: Running, Role: LEADER
I20260928 22:48:54.677937 16958 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88ee8fd3b6b044349626236bb83cdb62 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:48:54.678272 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 16890
I20260928 22:48:54.678396 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:48:54.679147 16961 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88ee8fd3b6b044349626236bb83cdb62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88ee8fd3b6b044349626236bb83cdb62" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 42783 } }
I20260928 22:48:54.679415 16870 external_mini_cluster.cc:1428] Setting key 944986af28187267332b19ce40f30464
I20260928 22:48:54.679986 16962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88ee8fd3b6b044349626236bb83cdb62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88ee8fd3b6b044349626236bb83cdb62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88ee8fd3b6b044349626236bb83cdb62" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 42783 } } }
I20260928 22:48:54.679979 16963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88ee8fd3b6b044349626236bb83cdb62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88ee8fd3b6b044349626236bb83cdb62. Latest consensus state: current_term: 1 leader_uuid: "88ee8fd3b6b044349626236bb83cdb62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88ee8fd3b6b044349626236bb83cdb62" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 42783 } } }
I20260928 22:48:54.680286 16963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88ee8fd3b6b044349626236bb83cdb62 [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:54.680282 16962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88ee8fd3b6b044349626236bb83cdb62 [sys.catalog]: This master's current role is: LEADER
W20260928 22:48:54.680750 16975 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 88ee8fd3b6b044349626236bb83cdb62: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:48:54.680853 16975 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:48:54.680962 16976 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:48:54.681157 16976 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:48:54.682973 16976 catalog_manager.cc:1347] Generated new cluster ID: c4f05c672fcd4746b4719100019c4a5b
I20260928 22:48:54.683037 16976 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 22:48:54 dist-test-slave-56vw krb5kdc[16877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:48:54.700784 16980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:54.680967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57790 (local address 127.16.121.190:42783)
0928 22:48:54.682351 (+ 1384us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:54.682360 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:54.682407 (+ 47us) server_negotiation.cc:407] Connection header received
0928 22:48:54.684047 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:54.684055 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:54.684123 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:54.684281 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:54.686382 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:54.688565 (+ 2183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:54.690225 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:54.690933 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:54.691103 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:54.696119 (+ 5016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:54.696126 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:54.696126 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:54.696142 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:54.698384 (+ 2242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:54.699017 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:54.699020 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:54.699020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:54.699178 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:54.699523 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:54.699523 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:54.699524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:54.699876 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:54.700041 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:54.700383 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 22:48:54.700537 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1322,"thread_start_us":78,"threads_started":1}
I20260928 22:48:54.705058 16976 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:48:54.705842 16976 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:48:54.713151 16976 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88ee8fd3b6b044349626236bb83cdb62: Generated new TSK 0
I20260928 22:48:54.713415 16976 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.121.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab.
Entry for principal kudu/127.16.121.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab.
I20260928 22:48:54.747694 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.129:0
--local_ip_for_outbound_sockets=127.16.121.129
--webserver_interface=127.16.121.129
--webserver_port=0
--tserver_master_addrs=127.16.121.190:42783
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab
--principal=kudu/127.16.121.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:36543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:48:54.832082 16984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:54.832327 16984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:54.832369 16984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:54.833600 16984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:54.833663 16984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:54.833709 16984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.129
I20260928 22:48:54.835325 16984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:36543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab
--principal=kudu/127.16.121.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.121.129
--webserver_port=0
--tserver_master_addrs=127.16.121.190:42783
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.129
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:54.835613 16984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:54.835853 16984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:54.841043 16989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:54.840996 16992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:54.840996 16990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:54.841456 16984 server_base.cc:1034] running on GCE node
I20260928 22:48:54.841647 16984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:54.841847 16984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:54.843036 16984 hybrid_clock.cc:648] HybridClock initialized: now 1790635734843021 us; error 33 us; skew 500 ppm
Sep 28 22:48:54 dist-test-slave-56vw krb5kdc[16877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635734, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:48:54.845777 16984 init.cc:377] Logged in from keytab as kudu/127.16.121.129@KRBTEST.COM (short username kudu)
I20260928 22:48:54.846199 16984 webserver.cc:460] Webserver started at http://127.16.121.129:39127/ using document root <none> and password file <none>
I20260928 22:48:54.846398 16984 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:54.846439 16984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:54.846529 16984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:54.847625 16984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-0/data/instance:
uuid: "7b46d6bef91e479b95df3abfc6be9c44"
format_stamp: "Formatted at 2026-09-28 22:48:54 on dist-test-slave-56vw"
server_key: "752d55b948351a965aafeffffb04887d"
server_key_iv: "86b876795bc82d5a56483ff9d3a3cff3"
server_key_version: "encryptionkey@0"
I20260928 22:48:54.847944 16984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance:
uuid: "7b46d6bef91e479b95df3abfc6be9c44"
format_stamp: "Formatted at 2026-09-28 22:48:54 on dist-test-slave-56vw"
server_key: "752d55b948351a965aafeffffb04887d"
server_key_iv: "86b876795bc82d5a56483ff9d3a3cff3"
server_key_version: "encryptionkey@0"
I20260928 22:48:54.849329 16984 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:48:54.850291 16999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:54.850642 16984 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:48:54.850718 16984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-0/wal
uuid: "7b46d6bef91e479b95df3abfc6be9c44"
format_stamp: "Formatted at 2026-09-28 22:48:54 on dist-test-slave-56vw"
server_key: "752d55b948351a965aafeffffb04887d"
server_key_iv: "86b876795bc82d5a56483ff9d3a3cff3"
server_key_version: "encryptionkey@0"
I20260928 22:48:54.850826 16984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:54.865181 16984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:54.867976 16984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:54.868283 16984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:54.868564 16984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:48:54.868939 16984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:48:54.869010 16984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:54.869056 16984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:48:54.869087 16984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:54.875859 16984 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.129:39601
I20260928 22:48:54.875905 17112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.129:39601 every 8 connection(s)
I20260928 22:48:54.876289 16984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
Sep 28 22:48:54 dist-test-slave-56vw krb5kdc[16877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635734, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:48:54.883839 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 16984
I20260928 22:48:54.883929 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance
I20260928 22:48:54.884141 16870 external_mini_cluster.cc:1428] Setting key 5f077f93621f30bc7085c5d5d12ea257
I20260928 22:48:54.887383 16980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:54.877144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:40857 (local address 127.16.121.190:42783)
0928 22:48:54.877213 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:54.877215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:54.877477 (+ 262us) server_negotiation.cc:407] Connection header received
0928 22:48:54.878043 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:54.878044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:54.878061 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:54.878117 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:54.879372 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:54.880020 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:54.881145 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:54.881394 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:54.881487 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:54.884099 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:54.884101 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:54.884101 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:54.884110 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:54.885749 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:54.886250 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:54.886252 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:54.886252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:54.886318 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:54.886612 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:54.886613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:54.886614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:54.886748 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:54.886829 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:54.887174 (+ 345us) server_negotiation.cc:299] Negotiation successful
0928 22:48:54.887280 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 22:48:54.887611 17115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:54.877132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:42783 (local address 127.16.121.129:40857)
0928 22:48:54.877306 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:54.877354 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:54.877869 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:54.878211 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:54.878215 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:54.878948 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:54.879269 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:54.879275 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:54.880118 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:54.880120 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:54.881041 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:54.881044 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:54.881498 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:54.881499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:54.881641 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:54.882291 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:48:54.882301 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:54.883945 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:54.885874 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:54.885882 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:54.885885 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:54.886155 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:54.886398 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:54.886399 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:54.886399 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:54.886529 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:54.886819 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:54.886826 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:54.886989 (+ 163us) client_negotiation.cc:769] Sending connection context
0928 22:48:54.887150 (+ 161us) client_negotiation.cc:240] Negotiation successful
0928 22:48:54.887335 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":97,"threads_started":1}
I20260928 22:48:54.888183 17113 heartbeater.cc:347] Connected to a master server at 127.16.121.190:42783
I20260928 22:48:54.888319 17113 heartbeater.cc:464] Registering TS with master...
I20260928 22:48:54.888581 17113 heartbeater.cc:511] Master 127.16.121.190:42783 requested a full tablet report, sending...
I20260928 22:48:54.889278 16922 ts_manager.cc:194] Registered new tserver with Master: 7b46d6bef91e479b95df3abfc6be9c44 (127.16.121.129:39601)
I20260928 22:48:54.890429 16922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.129@KRBTEST.COM'} at 127.16.121.129:40857
WARNING: no policy specified for kudu/127.16.121.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab.
Entry for principal kudu/127.16.121.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab.
I20260928 22:48:54.914743 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.130:0
--local_ip_for_outbound_sockets=127.16.121.130
--webserver_interface=127.16.121.130
--webserver_port=0
--tserver_master_addrs=127.16.121.190:42783
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab
--principal=kudu/127.16.121.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:36543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:48:55.000486 17118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:55.000730 17118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:55.000757 17118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:55.001922 17118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:55.001960 17118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:55.001987 17118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.130
I20260928 22:48:55.003468 17118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:36543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab
--principal=kudu/127.16.121.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.121.130
--webserver_port=0
--tserver_master_addrs=127.16.121.190:42783
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.130
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:55.003662 17118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:55.003844 17118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:55.009315 17126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:55.009225 17123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:55.009225 17124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:55.009433 17118 server_base.cc:1034] running on GCE node
I20260928 22:48:55.009734 17118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:55.010046 17118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:55.011204 17118 hybrid_clock.cc:648] HybridClock initialized: now 1790635735011196 us; error 43 us; skew 500 ppm
Sep 28 22:48:55 dist-test-slave-56vw krb5kdc[16877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635735, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:48:55.013782 17118 init.cc:377] Logged in from keytab as kudu/127.16.121.130@KRBTEST.COM (short username kudu)
I20260928 22:48:55.014250 17118 webserver.cc:460] Webserver started at http://127.16.121.130:37019/ using document root <none> and password file <none>
I20260928 22:48:55.014456 17118 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:55.014495 17118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:55.014587 17118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:55.015859 17118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-1/data/instance:
uuid: "1bc3eca7f78d46ce8d906a53fc48278a"
format_stamp: "Formatted at 2026-09-28 22:48:55 on dist-test-slave-56vw"
server_key: "d3ffa9c1e5e9a4b1dd119486dc038f22"
server_key_iv: "ca1e195bfd5c3ca5c93f503f41226924"
server_key_version: "encryptionkey@0"
I20260928 22:48:55.016215 17118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance:
uuid: "1bc3eca7f78d46ce8d906a53fc48278a"
format_stamp: "Formatted at 2026-09-28 22:48:55 on dist-test-slave-56vw"
server_key: "d3ffa9c1e5e9a4b1dd119486dc038f22"
server_key_iv: "ca1e195bfd5c3ca5c93f503f41226924"
server_key_version: "encryptionkey@0"
I20260928 22:48:55.017575 17118 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:48:55.018321 17133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:55.018519 17118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:48:55.018606 17118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-1/wal
uuid: "1bc3eca7f78d46ce8d906a53fc48278a"
format_stamp: "Formatted at 2026-09-28 22:48:55 on dist-test-slave-56vw"
server_key: "d3ffa9c1e5e9a4b1dd119486dc038f22"
server_key_iv: "ca1e195bfd5c3ca5c93f503f41226924"
server_key_version: "encryptionkey@0"
I20260928 22:48:55.018664 17118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:55.037417 17118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:55.039853 17118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:55.039996 17118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:55.040233 17118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:48:55.040673 17118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:48:55.040711 17118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:55.040735 17118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:48:55.040747 17118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:55.046903 17118 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.130:46751
I20260928 22:48:55.047098 17246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.130:46751 every 8 connection(s)
I20260928 22:48:55.047410 17118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
I20260928 22:48:55.050714 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 17118
I20260928 22:48:55.050822 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance
I20260928 22:48:55.051012 16870 external_mini_cluster.cc:1428] Setting key f9d583ebcfc38e9bf73bbeacf629a508
Sep 28 22:48:55 dist-test-slave-56vw krb5kdc[16877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635735, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.130@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:48:55.059517 16980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:55.048334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:58767 (local address 127.16.121.190:42783)
0928 22:48:55.048384 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:55.048386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:55.048572 (+ 186us) server_negotiation.cc:407] Connection header received
0928 22:48:55.049103 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:55.049105 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:55.049141 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:55.049222 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:55.050245 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:55.051198 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:55.052639 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:55.052999 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:55.053079 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:55.055989 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:55.055991 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:55.055992 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:55.056002 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:55.057822 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:55.058278 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:55.058280 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:55.058280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:55.058347 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:55.058619 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:55.058620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:55.058620 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:55.058783 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:55.058866 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:55.059275 (+ 409us) server_negotiation.cc:299] Negotiation successful
0928 22:48:55.059376 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 22:48:55.059557 17249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:55.048234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:42783 (local address 127.16.121.130:58767)
0928 22:48:55.048397 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:55.048452 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:55.048959 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:55.049288 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:55.049292 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:55.049907 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:55.050119 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:55.050124 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:55.051323 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:55.051324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:55.052541 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:55.052544 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:55.053129 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:55.053131 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:55.053337 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:55.054081 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:48:55.054094 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:55.055870 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:55.057925 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:55.057932 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:55.057934 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:55.058198 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:55.058422 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:55.058422 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:55.058423 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:55.058551 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:55.058901 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:55.058908 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:55.059105 (+ 197us) client_negotiation.cc:769] Sending connection context
0928 22:48:55.059229 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 22:48:55.059376 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
I20260928 22:48:55.060120 17247 heartbeater.cc:347] Connected to a master server at 127.16.121.190:42783
I20260928 22:48:55.060233 17247 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.16.121.131@KRBTEST.COM; defaulting to no policy
I20260928 22:48:55.060456 17247 heartbeater.cc:511] Master 127.16.121.190:42783 requested a full tablet report, sending...
I20260928 22:48:55.060878 16922 ts_manager.cc:194] Registered new tserver with Master: 1bc3eca7f78d46ce8d906a53fc48278a (127.16.121.130:46751)
I20260928 22:48:55.061288 16922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.130@KRBTEST.COM'} at 127.16.121.130:58767
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab.
Entry for principal kudu/127.16.121.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab.
I20260928 22:48:55.082191 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.131:0
--local_ip_for_outbound_sockets=127.16.121.131
--webserver_interface=127.16.121.131
--webserver_port=0
--tserver_master_addrs=127.16.121.190:42783
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab
--principal=kudu/127.16.121.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:36543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:48:55.171593 17252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:55.171828 17252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:55.171870 17252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:55.173101 17252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:55.173202 17252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:55.173252 17252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.131
I20260928 22:48:55.174777 17252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:36543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab
--principal=kudu/127.16.121.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.121.131
--webserver_port=0
--tserver_master_addrs=127.16.121.190:42783
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.131
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:55.175097 17252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:55.175344 17252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:55.181071 17257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:55.181066 17258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:55.181277 17252 server_base.cc:1034] running on GCE node
W20260928 22:48:55.181067 17260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:55.181564 17252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:55.181813 17252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:55.182979 17252 hybrid_clock.cc:648] HybridClock initialized: now 1790635735182973 us; error 39 us; skew 500 ppm
Sep 28 22:48:55 dist-test-slave-56vw krb5kdc[16877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635735, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:48:55.185866 17252 init.cc:377] Logged in from keytab as kudu/127.16.121.131@KRBTEST.COM (short username kudu)
I20260928 22:48:55.186380 17252 webserver.cc:460] Webserver started at http://127.16.121.131:43441/ using document root <none> and password file <none>
I20260928 22:48:55.186666 17252 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:55.186823 17252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:55.187009 17252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:55.188300 17252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-2/data/instance:
uuid: "4c9b1e1984e841b0a30aaba4004ff18e"
format_stamp: "Formatted at 2026-09-28 22:48:55 on dist-test-slave-56vw"
server_key: "960e3950b93a419969bab8dbfc2f949b"
server_key_iv: "fee63a371d4a5b5d41ec94e47c1baa76"
server_key_version: "encryptionkey@0"
I20260928 22:48:55.188689 17252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance:
uuid: "4c9b1e1984e841b0a30aaba4004ff18e"
format_stamp: "Formatted at 2026-09-28 22:48:55 on dist-test-slave-56vw"
server_key: "960e3950b93a419969bab8dbfc2f949b"
server_key_iv: "fee63a371d4a5b5d41ec94e47c1baa76"
server_key_version: "encryptionkey@0"
I20260928 22:48:55.190522 17252 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260928 22:48:55.191428 17267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:55.191617 17252 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:48:55.191720 17252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-2/wal
uuid: "4c9b1e1984e841b0a30aaba4004ff18e"
format_stamp: "Formatted at 2026-09-28 22:48:55 on dist-test-slave-56vw"
server_key: "960e3950b93a419969bab8dbfc2f949b"
server_key_iv: "fee63a371d4a5b5d41ec94e47c1baa76"
server_key_version: "encryptionkey@0"
I20260928 22:48:55.191792 17252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:55.202005 17252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:55.204172 17252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:55.204336 17252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:55.204629 17252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:48:55.204937 17252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:48:55.204986 17252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:55.205026 17252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:48:55.205050 17252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:55.211133 17252 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.131:37427
I20260928 22:48:55.211196 17380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.131:37427 every 8 connection(s)
I20260928 22:48:55.211544 17252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
I20260928 22:48:55.217726 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 17252
I20260928 22:48:55.217841 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance
I20260928 22:48:55.218012 16870 external_mini_cluster.cc:1428] Setting key bc24137a93106bb3439092f1d605beb1
Sep 28 22:48:55 dist-test-slave-56vw krb5kdc[16877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635735, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.131@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:48:55.222221 16980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:55.212316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:55841 (local address 127.16.121.190:42783)
0928 22:48:55.212383 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:55.212385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:55.212652 (+ 267us) server_negotiation.cc:407] Connection header received
0928 22:48:55.213349 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:55.213350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:55.213361 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:55.213493 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:55.214520 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:55.215150 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:55.216191 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:55.216476 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:55.216543 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:55.219026 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:55.219028 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:55.219028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:55.219035 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:55.220571 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:55.221002 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:55.221003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:55.221004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:55.221083 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:55.221478 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:55.221479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:55.221479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:55.221620 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:55.221690 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:55.222051 (+ 361us) server_negotiation.cc:299] Negotiation successful
0928 22:48:55.222140 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 22:48:55.222488 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:55.212310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:42783 (local address 127.16.121.131:55841)
0928 22:48:55.212520 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:55.212560 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:55.213255 (+ 695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:55.213577 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:55.213581 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:55.214187 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:55.214385 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:55.214389 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:55.215231 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:55.215232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:55.216076 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:55.216078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:55.216563 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:55.216564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:55.216684 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:55.217241 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:48:55.217250 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:55.218933 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:55.220660 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:55.220666 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:55.220668 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:55.220906 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:55.221178 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:55.221179 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:55.221180 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:55.221300 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:55.221707 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:55.221713 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:55.221877 (+ 164us) client_negotiation.cc:769] Sending connection context
0928 22:48:55.221998 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 22:48:55.222212 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":100,"threads_started":1}
I20260928 22:48:55.223192 17381 heartbeater.cc:347] Connected to a master server at 127.16.121.190:42783
I20260928 22:48:55.223344 17381 heartbeater.cc:464] Registering TS with master...
I20260928 22:48:55.223541 17381 heartbeater.cc:511] Master 127.16.121.190:42783 requested a full tablet report, sending...
I20260928 22:48:55.224020 16922 ts_manager.cc:194] Registered new tserver with Master: 4c9b1e1984e841b0a30aaba4004ff18e (127.16.121.131:37427)
I20260928 22:48:55.224525 16922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.131@KRBTEST.COM'} at 127.16.121.131:55841
I20260928 22:48:55.230525 16870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 22:48:55 dist-test-slave-56vw krb5kdc[16877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635735, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 22:48:55 dist-test-slave-56vw krb5kdc[16877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635735, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.121.129@KRBTEST.COM
I20260928 22:48:55.256134 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:55.246949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45778 (local address 127.16.121.129:39601)
0928 22:48:55.247130 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:55.247140 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:55.247159 (+ 19us) server_negotiation.cc:407] Connection header received
0928 22:48:55.247188 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:55.247190 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:55.247204 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:55.247286 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:55.247877 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:55.248720 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:55.249629 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:55.250059 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:55.250152 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:55.252646 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:55.252656 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:55.252656 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:55.252664 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:55.254712 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:55.255162 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:55.255165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:55.255166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:55.255239 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:55.255523 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:55.255524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:55.255524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:55.255694 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:55.255814 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:55.255959 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 22:48:55.256049 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":86,"threads_started":1}
W20260928 22:48:55.256521 17027 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45778
Sep 28 22:48:55 dist-test-slave-56vw krb5kdc[16877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 22:48:55 dist-test-slave-56vw krb5kdc[16877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.121.129@KRBTEST.COM
I20260928 22:48:55.280547 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:55.272427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45788 (local address 127.16.121.129:39601)
0928 22:48:55.272490 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:55.272492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:55.272643 (+ 151us) server_negotiation.cc:407] Connection header received
0928 22:48:55.272701 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:55.272702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:55.272712 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:55.272780 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:55.273436 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:55.273968 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:55.274711 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:55.274944 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:55.275004 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:55.277652 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:55.277654 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:55.277654 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:55.277662 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:55.279235 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:55.279669 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:55.279670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:55.279670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:55.279729 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:55.280021 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:55.280022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:55.280022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:55.280141 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:55.280249 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:55.280348 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 22:48:55.280477 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 22:48:55.281162 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 16984
I20260928 22:48:55.286942 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 17118
I20260928 22:48:55.291747 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 17252
I20260928 22:48:55.296247 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 16890
2026-09-28T22:48:55Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1974 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:48:55 dist-test-slave-56vw krb5kdc[17395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:48:55 dist-test-slave-56vw krb5kdc[17395](info): set up 2 sockets
Sep 28 22:48:55 dist-test-slave-56vw krb5kdc[17395](info): commencing operation
krb5kdc: starting...
W20260928 22:48:56.390588 16870 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Sep 28 22:48:56 dist-test-slave-56vw krb5kdc[17395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:48:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:48:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.121.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
I20260928 22:48:56.515540 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:44003
--webserver_interface=127.16.121.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:38099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:44003
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:48:56.602239 17408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:56.602532 17408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:56.602571 17408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:56.603885 17408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:48:56.603953 17408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:56.603977 17408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:56.604007 17408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:48:56.604035 17408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:48:56.605569 17408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:38099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:44003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:44003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.121.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:56.605849 17408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:56.606083 17408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:56.611017 17416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:56.611039 17413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:56.611181 17408 server_base.cc:1034] running on GCE node
W20260928 22:48:56.611023 17414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:56.611632 17408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:56.611898 17408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:56.613040 17408 hybrid_clock.cc:648] HybridClock initialized: now 1790635736613029 us; error 29 us; skew 500 ppm
Sep 28 22:48:56 dist-test-slave-56vw krb5kdc[17395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635736, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:48:56.615684 17408 init.cc:377] Logged in from keytab as kudu/127.16.121.190@KRBTEST.COM (short username kudu)
I20260928 22:48:56.616117 17408 webserver.cc:460] Webserver started at http://127.16.121.190:39821/ using document root <none> and password file <none>
I20260928 22:48:56.616353 17408 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:56.616401 17408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:56.616616 17408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:56.617715 17408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "ef18c23dca294f9aac346fb9c7ef59d2"
format_stamp: "Formatted at 2026-09-28 22:48:56 on dist-test-slave-56vw"
server_key: "359155ebcddba4005f3de8d874613a25"
server_key_iv: "6a5558c4a9fb7fd61043bbb5fb452711"
server_key_version: "encryptionkey@0"
I20260928 22:48:56.618088 17408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "ef18c23dca294f9aac346fb9c7ef59d2"
format_stamp: "Formatted at 2026-09-28 22:48:56 on dist-test-slave-56vw"
server_key: "359155ebcddba4005f3de8d874613a25"
server_key_iv: "6a5558c4a9fb7fd61043bbb5fb452711"
server_key_version: "encryptionkey@0"
I20260928 22:48:56.619721 17408 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:48:56.620703 17423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:56.620883 17408 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:48:56.620972 17408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "ef18c23dca294f9aac346fb9c7ef59d2"
format_stamp: "Formatted at 2026-09-28 22:48:56 on dist-test-slave-56vw"
server_key: "359155ebcddba4005f3de8d874613a25"
server_key_iv: "6a5558c4a9fb7fd61043bbb5fb452711"
server_key_version: "encryptionkey@0"
I20260928 22:48:56.621032 17408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:56.631300 17408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:56.633371 17408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:56.633492 17408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:56.637362 17408 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.190:44003
I20260928 22:48:56.637418 17475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.190:44003 every 8 connection(s)
I20260928 22:48:56.637789 17408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:48:56.638799 17476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:56.641004 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 17408
I20260928 22:48:56.641103 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:48:56.641261 16870 external_mini_cluster.cc:1428] Setting key 1fbb7fc1e7f18e2a7517c2f25e4b100f
I20260928 22:48:56.641506 17476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef18c23dca294f9aac346fb9c7ef59d2: Bootstrap starting.
I20260928 22:48:56.642277 17476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef18c23dca294f9aac346fb9c7ef59d2: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:56.642606 17476 log.cc:824] T 00000000000000000000000000000000 P ef18c23dca294f9aac346fb9c7ef59d2: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:56.643748 17476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef18c23dca294f9aac346fb9c7ef59d2: No bootstrap required, opened a new log
I20260928 22:48:56.645941 17476 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef18c23dca294f9aac346fb9c7ef59d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef18c23dca294f9aac346fb9c7ef59d2" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 44003 } }
I20260928 22:48:56.646139 17476 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef18c23dca294f9aac346fb9c7ef59d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:56.646169 17476 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef18c23dca294f9aac346fb9c7ef59d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18c23dca294f9aac346fb9c7ef59d2, State: Initialized, Role: FOLLOWER
I20260928 22:48:56.646325 17476 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef18c23dca294f9aac346fb9c7ef59d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef18c23dca294f9aac346fb9c7ef59d2" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 44003 } }
I20260928 22:48:56.646411 17476 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef18c23dca294f9aac346fb9c7ef59d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:48:56.646453 17476 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef18c23dca294f9aac346fb9c7ef59d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:48:56.646503 17476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef18c23dca294f9aac346fb9c7ef59d2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:56.647351 17476 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef18c23dca294f9aac346fb9c7ef59d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef18c23dca294f9aac346fb9c7ef59d2" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 44003 } }
I20260928 22:48:56.647475 17476 leader_election.cc:302] T 00000000000000000000000000000000 P ef18c23dca294f9aac346fb9c7ef59d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef18c23dca294f9aac346fb9c7ef59d2; no voters:
I20260928 22:48:56.647675 17476 leader_election.cc:288] T 00000000000000000000000000000000 P ef18c23dca294f9aac346fb9c7ef59d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:48:56.647735 17481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef18c23dca294f9aac346fb9c7ef59d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:48:56.647935 17481 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef18c23dca294f9aac346fb9c7ef59d2 [term 1 LEADER]: Becoming Leader. State: Replica: ef18c23dca294f9aac346fb9c7ef59d2, State: Running, Role: LEADER
I20260928 22:48:56.647974 17476 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef18c23dca294f9aac346fb9c7ef59d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:48:56.648037 17481 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef18c23dca294f9aac346fb9c7ef59d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef18c23dca294f9aac346fb9c7ef59d2" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 44003 } }
I20260928 22:48:56.648355 17481 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef18c23dca294f9aac346fb9c7ef59d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef18c23dca294f9aac346fb9c7ef59d2. Latest consensus state: current_term: 1 leader_uuid: "ef18c23dca294f9aac346fb9c7ef59d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef18c23dca294f9aac346fb9c7ef59d2" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 44003 } } }
I20260928 22:48:56.648468 17481 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef18c23dca294f9aac346fb9c7ef59d2 [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:56.648375 17483 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef18c23dca294f9aac346fb9c7ef59d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef18c23dca294f9aac346fb9c7ef59d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef18c23dca294f9aac346fb9c7ef59d2" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 44003 } } }
I20260928 22:48:56.648638 17483 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef18c23dca294f9aac346fb9c7ef59d2 [sys.catalog]: This master's current role is: LEADER
Sep 28 22:48:56 dist-test-slave-56vw krb5kdc[17395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:48:56.649505 17488 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:48:56.650183 17488 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:48:56.651978 17488 catalog_manager.cc:1347] Generated new cluster ID: 2d3abf9d47a848a9b56454c7d4f14c89
I20260928 22:48:56.652032 17488 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:48:56.653873 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:56.641828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56380 (local address 127.16.121.190:44003)
0928 22:48:56.642036 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:56.642046 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:56.642081 (+ 35us) server_negotiation.cc:407] Connection header received
0928 22:48:56.642623 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:56.642632 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:56.642720 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:56.642856 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:56.643828 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:56.644854 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:56.646041 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:56.646707 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:56.646800 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:56.649888 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:56.649892 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:56.649893 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:56.649907 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:56.652142 (+ 2235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:56.652596 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:56.652599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:56.652600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:56.652704 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:56.652922 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:56.652923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:56.652923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:56.653237 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:56.653387 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:56.653523 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 22:48:56.653665 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":123,"threads_started":1}
I20260928 22:48:56.663241 17488 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:48:56.663765 17488 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:48:56.682133 17488 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef18c23dca294f9aac346fb9c7ef59d2: Generated new TSK 0
I20260928 22:48:56.682368 17488 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.121.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab.
Entry for principal kudu/127.16.121.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab.
I20260928 22:48:56.718461 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.129:0
--local_ip_for_outbound_sockets=127.16.121.129
--webserver_interface=127.16.121.129
--webserver_port=0
--tserver_master_addrs=127.16.121.190:44003
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab
--principal=kudu/127.16.121.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:38099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:48:56.822741 17502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:56.823004 17502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:56.823032 17502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:56.824412 17502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:56.824479 17502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:56.824512 17502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.129
I20260928 22:48:56.826565 17502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:38099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab
--principal=kudu/127.16.121.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.121.129
--webserver_port=0
--tserver_master_addrs=127.16.121.190:44003
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.129
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:56.826809 17502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:56.827011 17502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:56.832515 17510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:56.832564 17508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:56.832563 17507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:56.833045 17502 server_base.cc:1034] running on GCE node
I20260928 22:48:56.833274 17502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:56.833642 17502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:56.834873 17502 hybrid_clock.cc:648] HybridClock initialized: now 1790635736834857 us; error 34 us; skew 500 ppm
Sep 28 22:48:56 dist-test-slave-56vw krb5kdc[17395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635736, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:48:56.837877 17502 init.cc:377] Logged in from keytab as kudu/127.16.121.129@KRBTEST.COM (short username kudu)
I20260928 22:48:56.838420 17502 webserver.cc:460] Webserver started at http://127.16.121.129:39699/ using document root <none> and password file <none>
I20260928 22:48:56.838675 17502 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:56.838745 17502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:56.838887 17502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:56.840154 17502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-0/data/instance:
uuid: "5558ab0a8f904298adadbae09773156c"
format_stamp: "Formatted at 2026-09-28 22:48:56 on dist-test-slave-56vw"
server_key: "5c5ca3210a4bc2c9156230d97c3ccb3d"
server_key_iv: "973a448632e830bf097ab6f889815525"
server_key_version: "encryptionkey@0"
I20260928 22:48:56.840536 17502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance:
uuid: "5558ab0a8f904298adadbae09773156c"
format_stamp: "Formatted at 2026-09-28 22:48:56 on dist-test-slave-56vw"
server_key: "5c5ca3210a4bc2c9156230d97c3ccb3d"
server_key_iv: "973a448632e830bf097ab6f889815525"
server_key_version: "encryptionkey@0"
I20260928 22:48:56.841935 17502 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:48:56.842742 17517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:56.842942 17502 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:48:56.843041 17502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-0/wal
uuid: "5558ab0a8f904298adadbae09773156c"
format_stamp: "Formatted at 2026-09-28 22:48:56 on dist-test-slave-56vw"
server_key: "5c5ca3210a4bc2c9156230d97c3ccb3d"
server_key_iv: "973a448632e830bf097ab6f889815525"
server_key_version: "encryptionkey@0"
I20260928 22:48:56.843159 17502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:56.857427 17502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:56.859735 17502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:56.859913 17502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:56.860177 17502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:48:56.860515 17502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:48:56.860566 17502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:56.860601 17502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:48:56.860637 17502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:56.866670 17502 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.129:40363
I20260928 22:48:56.866750 17630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.129:40363 every 8 connection(s)
I20260928 22:48:56.867115 17502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
Sep 28 22:48:56 dist-test-slave-56vw krb5kdc[17395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635736, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:48:56.875303 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 17502
I20260928 22:48:56.875408 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance
I20260928 22:48:56.875595 16870 external_mini_cluster.cc:1428] Setting key 7676890b2061e8e33f481af35616e117
I20260928 22:48:56.877678 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:56.867950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:53953 (local address 127.16.121.190:44003)
0928 22:48:56.868048 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:56.868051 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:56.868234 (+ 183us) server_negotiation.cc:407] Connection header received
0928 22:48:56.869245 (+ 1011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:56.869246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:56.869258 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:56.869318 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:56.869991 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:56.870525 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:56.871654 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:56.871876 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:56.871963 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:56.874274 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:56.874276 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:56.874277 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:56.874285 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:56.876146 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:56.876614 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:56.876616 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:56.876616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:56.876690 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:56.876960 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:56.876961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:56.876961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:56.877081 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:56.877150 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:56.877530 (+ 380us) server_negotiation.cc:299] Negotiation successful
0928 22:48:56.877610 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 22:48:56.877964 17633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:56.867924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44003 (local address 127.16.121.129:53953)
0928 22:48:56.868099 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:56.868147 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:56.869127 (+ 980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:56.869396 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:56.869400 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:56.869679 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:56.869893 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:56.869897 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:56.870633 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:56.870634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:56.871558 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:56.871561 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:56.871973 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:56.871974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:56.872101 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:56.872656 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:48:56.872665 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:56.874160 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:56.876238 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:56.876248 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:56.876250 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:56.876513 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:56.876758 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:56.876758 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:56.876759 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:56.876887 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:56.877152 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:56.877160 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:56.877353 (+ 193us) client_negotiation.cc:769] Sending connection context
0928 22:48:56.877488 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 22:48:56.877682 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":81,"threads_started":1}
I20260928 22:48:56.878589 17631 heartbeater.cc:347] Connected to a master server at 127.16.121.190:44003
I20260928 22:48:56.878702 17631 heartbeater.cc:464] Registering TS with master...
I20260928 22:48:56.878894 17631 heartbeater.cc:511] Master 127.16.121.190:44003 requested a full tablet report, sending...
I20260928 22:48:56.879518 17440 ts_manager.cc:194] Registered new tserver with Master: 5558ab0a8f904298adadbae09773156c (127.16.121.129:40363)
I20260928 22:48:56.880503 17440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.129@KRBTEST.COM'} at 127.16.121.129:53953
WARNING: no policy specified for kudu/127.16.121.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab.
Entry for principal kudu/127.16.121.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab.
I20260928 22:48:56.902954 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.130:0
--local_ip_for_outbound_sockets=127.16.121.130
--webserver_interface=127.16.121.130
--webserver_port=0
--tserver_master_addrs=127.16.121.190:44003
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab
--principal=kudu/127.16.121.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:38099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:48:56.986790 17636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:56.987025 17636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:56.987073 17636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:56.988425 17636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:56.988476 17636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:56.988507 17636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.130
I20260928 22:48:56.990083 17636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:38099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab
--principal=kudu/127.16.121.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.121.130
--webserver_port=0
--tserver_master_addrs=127.16.121.190:44003
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.130
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:56.990360 17636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:56.990626 17636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:56.996090 17642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:56.996214 17641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:56.996181 17644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:56.996310 17636 server_base.cc:1034] running on GCE node
I20260928 22:48:56.996552 17636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:56.996765 17636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:56.997905 17636 hybrid_clock.cc:648] HybridClock initialized: now 1790635736997892 us; error 30 us; skew 500 ppm
Sep 28 22:48:56 dist-test-slave-56vw krb5kdc[17395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635736, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:48:57.000799 17636 init.cc:377] Logged in from keytab as kudu/127.16.121.130@KRBTEST.COM (short username kudu)
I20260928 22:48:57.001317 17636 webserver.cc:460] Webserver started at http://127.16.121.130:42801/ using document root <none> and password file <none>
I20260928 22:48:57.001546 17636 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:57.001593 17636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:57.001693 17636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:57.002877 17636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-1/data/instance:
uuid: "18bc355d264442c6932effb39d4b3514"
format_stamp: "Formatted at 2026-09-28 22:48:56 on dist-test-slave-56vw"
server_key: "e9010a3ee5a2532f770bad5d4f99d01c"
server_key_iv: "e70bcb6bf7e5bb7c7109e23b08fd9e65"
server_key_version: "encryptionkey@0"
I20260928 22:48:57.003275 17636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance:
uuid: "18bc355d264442c6932effb39d4b3514"
format_stamp: "Formatted at 2026-09-28 22:48:56 on dist-test-slave-56vw"
server_key: "e9010a3ee5a2532f770bad5d4f99d01c"
server_key_iv: "e70bcb6bf7e5bb7c7109e23b08fd9e65"
server_key_version: "encryptionkey@0"
I20260928 22:48:57.004773 17636 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:48:57.005650 17651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:57.005885 17636 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:48:57.005996 17636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-1/wal
uuid: "18bc355d264442c6932effb39d4b3514"
format_stamp: "Formatted at 2026-09-28 22:48:56 on dist-test-slave-56vw"
server_key: "e9010a3ee5a2532f770bad5d4f99d01c"
server_key_iv: "e70bcb6bf7e5bb7c7109e23b08fd9e65"
server_key_version: "encryptionkey@0"
I20260928 22:48:57.006088 17636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:57.013549 17636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:57.015902 17636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:57.016083 17636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:57.016335 17636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:48:57.016671 17636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:48:57.016724 17636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:57.016765 17636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:48:57.016798 17636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:57.023303 17636 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.130:37275
I20260928 22:48:57.023352 17764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.130:37275 every 8 connection(s)
I20260928 22:48:57.023728 17636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
I20260928 22:48:57.028374 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 17636
I20260928 22:48:57.028507 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance
I20260928 22:48:57.028703 16870 external_mini_cluster.cc:1428] Setting key c32b2014cf8879055d21877765b3fa36
Sep 28 22:48:57 dist-test-slave-56vw krb5kdc[17395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635736, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.130@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:48:57.035392 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:57.024489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:35017 (local address 127.16.121.190:44003)
0928 22:48:57.024613 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:57.024616 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:57.024854 (+ 238us) server_negotiation.cc:407] Connection header received
0928 22:48:57.025851 (+ 997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:57.025852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:57.025863 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:57.025939 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:57.026584 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:57.027144 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:57.028463 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:57.028856 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:57.028977 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:57.031862 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:57.031864 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:57.031865 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:57.031873 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:57.033647 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:57.034082 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:57.034084 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:57.034085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:57.034154 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:57.034526 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:57.034527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:57.034528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:57.034698 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:57.034780 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:57.035179 (+ 399us) server_negotiation.cc:299] Negotiation successful
0928 22:48:57.035276 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260928 22:48:57.035532 17767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:57.024491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44003 (local address 127.16.121.130:35017)
0928 22:48:57.024720 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:57.024764 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:57.025724 (+ 960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:57.025997 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:57.026001 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:57.026278 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:57.026483 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:57.026488 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:57.027417 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:57.027419 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:57.028358 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:57.028360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:57.028996 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:57.028997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:57.029143 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:57.029745 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:48:57.029762 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:57.031743 (+ 1981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:57.033734 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:57.033740 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:57.033743 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:57.034008 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:57.034243 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:57.034245 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:57.034245 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:57.034439 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:57.034783 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:57.034793 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:57.034981 (+ 188us) client_negotiation.cc:769] Sending connection context
0928 22:48:57.035146 (+ 165us) client_negotiation.cc:240] Negotiation successful
0928 22:48:57.035298 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":148,"threads_started":1}
I20260928 22:48:57.036275 17765 heartbeater.cc:347] Connected to a master server at 127.16.121.190:44003
I20260928 22:48:57.036419 17765 heartbeater.cc:464] Registering TS with master...
I20260928 22:48:57.036612 17765 heartbeater.cc:511] Master 127.16.121.190:44003 requested a full tablet report, sending...
I20260928 22:48:57.037091 17440 ts_manager.cc:194] Registered new tserver with Master: 18bc355d264442c6932effb39d4b3514 (127.16.121.130:37275)
I20260928 22:48:57.037650 17440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.130@KRBTEST.COM'} at 127.16.121.130:35017
WARNING: no policy specified for kudu/127.16.121.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab.
Entry for principal kudu/127.16.121.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab.
I20260928 22:48:57.058394 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.131:0
--local_ip_for_outbound_sockets=127.16.121.131
--webserver_interface=127.16.121.131
--webserver_port=0
--tserver_master_addrs=127.16.121.190:44003
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab
--principal=kudu/127.16.121.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:38099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:48:57.141263 17770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:57.141497 17770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:57.141525 17770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:57.142735 17770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:57.142781 17770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:57.142807 17770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.131
I20260928 22:48:57.144408 17770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:38099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab
--principal=kudu/127.16.121.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.121.131
--webserver_port=0
--tserver_master_addrs=127.16.121.190:44003
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.131
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:57.144660 17770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:57.144898 17770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:57.150704 17775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:57.150722 17776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:57.150704 17778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:57.151095 17770 server_base.cc:1034] running on GCE node
I20260928 22:48:57.151331 17770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:57.151585 17770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:57.152781 17770 hybrid_clock.cc:648] HybridClock initialized: now 1790635737152753 us; error 40 us; skew 500 ppm
Sep 28 22:48:57 dist-test-slave-56vw krb5kdc[17395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635737, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:48:57.155555 17770 init.cc:377] Logged in from keytab as kudu/127.16.121.131@KRBTEST.COM (short username kudu)
I20260928 22:48:57.156026 17770 webserver.cc:460] Webserver started at http://127.16.121.131:45825/ using document root <none> and password file <none>
I20260928 22:48:57.156266 17770 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:57.156333 17770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:57.156527 17770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:57.157650 17770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-2/data/instance:
uuid: "fc4631a0d2364673a7e3931e209661dd"
format_stamp: "Formatted at 2026-09-28 22:48:57 on dist-test-slave-56vw"
server_key: "a1e50e135ee3952eecc87840ca3a586e"
server_key_iv: "06933e54320e29205b639ff826047eb0"
server_key_version: "encryptionkey@0"
I20260928 22:48:57.158028 17770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance:
uuid: "fc4631a0d2364673a7e3931e209661dd"
format_stamp: "Formatted at 2026-09-28 22:48:57 on dist-test-slave-56vw"
server_key: "a1e50e135ee3952eecc87840ca3a586e"
server_key_iv: "06933e54320e29205b639ff826047eb0"
server_key_version: "encryptionkey@0"
I20260928 22:48:57.159598 17770 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:48:57.160526 17785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:57.160741 17770 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:48:57.160840 17770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-2/wal
uuid: "fc4631a0d2364673a7e3931e209661dd"
format_stamp: "Formatted at 2026-09-28 22:48:57 on dist-test-slave-56vw"
server_key: "a1e50e135ee3952eecc87840ca3a586e"
server_key_iv: "06933e54320e29205b639ff826047eb0"
server_key_version: "encryptionkey@0"
I20260928 22:48:57.160926 17770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:57.181181 17770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:57.183463 17770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:57.183629 17770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:57.183866 17770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:48:57.184171 17770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:48:57.184222 17770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:57.184264 17770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:48:57.184291 17770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:57.190285 17770 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.131:40385
I20260928 22:48:57.190331 17898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.131:40385 every 8 connection(s)
I20260928 22:48:57.190656 17770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
I20260928 22:48:57.193812 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 17770
I20260928 22:48:57.193924 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance
I20260928 22:48:57.194098 16870 external_mini_cluster.cc:1428] Setting key 8bcf243974c9bf04c6e2526ae0107244
Sep 28 22:48:57 dist-test-slave-56vw krb5kdc[17395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635737, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.131@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:48:57.201453 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:57.191444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:48297 (local address 127.16.121.190:44003)
0928 22:48:57.191508 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:57.191510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:57.191722 (+ 212us) server_negotiation.cc:407] Connection header received
0928 22:48:57.192737 (+ 1015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:57.192738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:57.192750 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:57.192824 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:57.193628 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:57.194444 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:57.195688 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:57.195913 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:57.196012 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:57.198327 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:57.198329 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:57.198329 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:57.198339 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:57.199809 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:57.200340 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:57.200341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:57.200342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:57.200410 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:57.200722 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:57.200723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:57.200723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:57.200852 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:57.200935 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:57.201252 (+ 317us) server_negotiation.cc:299] Negotiation successful
0928 22:48:57.201342 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 22:48:57.201711 17901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:57.191409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44003 (local address 127.16.121.131:48297)
0928 22:48:57.191565 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:57.191608 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:57.192624 (+ 1016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:57.192885 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:57.192889 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:57.193146 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:57.193352 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:57.193356 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:57.194588 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:57.194589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:57.195577 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:57.195580 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:57.196024 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:57.196025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:57.196164 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:57.196747 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:48:57.196757 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:57.198223 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:57.199967 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:57.199974 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:57.199977 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:57.200237 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:57.200517 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:57.200518 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:57.200519 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:57.200640 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:57.200951 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:57.200958 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:57.201102 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 22:48:57.201231 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 22:48:57.201430 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
I20260928 22:48:57.202301 17899 heartbeater.cc:347] Connected to a master server at 127.16.121.190:44003
I20260928 22:48:57.202404 17899 heartbeater.cc:464] Registering TS with master...
I20260928 22:48:57.202555 17899 heartbeater.cc:511] Master 127.16.121.190:44003 requested a full tablet report, sending...
I20260928 22:48:57.202924 17440 ts_manager.cc:194] Registered new tserver with Master: fc4631a0d2364673a7e3931e209661dd (127.16.121.131:40385)
I20260928 22:48:57.203374 17440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.131@KRBTEST.COM'} at 127.16.121.131:48297
I20260928 22:48:57.205760 16870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:48:57.213773 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:57.207294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56394 (local address 127.16.121.190:44003)
0928 22:48:57.207432 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:57.207433 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:57.207461 (+ 28us) server_negotiation.cc:407] Connection header received
0928 22:48:57.207539 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:57.207540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:57.207550 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:57.207605 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:57.208170 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:57.208660 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:57.209364 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:57.209625 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:57.209697 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:57.210977 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:57.210978 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:57.210978 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:57.210986 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:57.212504 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:57.212906 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:57.212907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:57.212908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:57.212965 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:57.213170 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:57.213171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:57.213171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:57.213333 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:57.213438 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:57.213574 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 22:48:57.213663 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 22:48:57.216284 17440 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56394:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:48:57.218163 17440 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:48:57.225320 17914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:57.220995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.130:37275 (local address 127.0.0.1:40120)
0928 22:48:57.221282 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:57.221292 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:57.221346 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:57.221704 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:57.221705 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:57.221706 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:48:57.221785 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:57.221787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:57.223152 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:57.223153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:57.224229 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:57.224231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:57.225033 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:57.225034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:57.225098 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:57.225103 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:48:57.225161 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 22:48:57.225205 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":54,"thread_start_us":86,"threads_started":1}
I20260928 22:48:57.225315 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:57.221117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40120 (local address 127.16.121.130:37275)
0928 22:48:57.221303 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:57.221308 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:57.221410 (+ 102us) server_negotiation.cc:407] Connection header received
0928 22:48:57.221534 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:57.221537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:57.221556 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:57.221625 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:48:57.221868 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:57.223039 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:57.224319 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:57.224961 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:57.225034 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:57.225087 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:57.225179 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 22:48:57.225219 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":79,"threads_started":1}
I20260928 22:48:57.225580 17910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:57.220766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:40363 (local address 127.0.0.1:42322)
0928 22:48:57.220889 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:57.220932 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:57.221028 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:57.221467 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:57.221470 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:57.221472 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:48:57.221544 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:57.221546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:57.222586 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:57.222587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:57.223862 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:57.223865 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:57.224950 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:57.224951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:57.225038 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:57.225053 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 22:48:57.225135 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 22:48:57.225197 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":77,"threads_started":1}
I20260928 22:48:57.225612 17911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:57.220886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42322 (local address 127.16.121.129:40363)
0928 22:48:57.221115 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:57.221120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:57.221133 (+ 13us) server_negotiation.cc:407] Connection header received
0928 22:48:57.221180 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:57.221182 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:57.221201 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:57.221293 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:48:57.221630 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:57.222476 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:57.223951 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:57.224746 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:57.224838 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:57.224896 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:57.225311 (+ 415us) server_negotiation.cc:299] Negotiation successful
0928 22:48:57.225358 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":112,"threads_started":1}
I20260928 22:48:57.225754 17915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:57.221277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33826 (local address 127.16.121.131:40385)
0928 22:48:57.221460 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:57.221464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:57.221473 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:48:57.221512 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:57.221514 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:57.221530 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:57.221610 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:48:57.221869 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:57.223021 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:57.224476 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:57.225207 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:57.225271 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:57.225321 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:57.225618 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 22:48:57.225671 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":83,"threads_started":1}
I20260928 22:48:57.225889 17912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:57.221171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.131:40385 (local address 127.0.0.1:33826)
0928 22:48:57.221357 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:57.221368 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:57.221419 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:57.221699 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:57.221700 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:57.221702 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:48:57.221784 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:57.221787 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:57.223172 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:57.223173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:57.224375 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:57.224378 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:57.225353 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:57.225354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:57.225417 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:57.225422 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:48:57.225475 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 22:48:57.225523 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":54,"thread_start_us":62,"threads_started":1}
I20260928 22:48:57.226378 17699 tablet_service.cc:1467] Processing CreateTablet for tablet 653fa7149d2e456aaab057aec70ae6f6 (DEFAULT_TABLE table=test-table [id=5fe9ed7c176348fc8da4302d28b28775]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:48:57.226531 17833 tablet_service.cc:1467] Processing CreateTablet for tablet 653fa7149d2e456aaab057aec70ae6f6 (DEFAULT_TABLE table=test-table [id=5fe9ed7c176348fc8da4302d28b28775]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:48:57.226755 17699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 653fa7149d2e456aaab057aec70ae6f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:57.226783 17833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 653fa7149d2e456aaab057aec70ae6f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:57.227535 17565 tablet_service.cc:1467] Processing CreateTablet for tablet 653fa7149d2e456aaab057aec70ae6f6 (DEFAULT_TABLE table=test-table [id=5fe9ed7c176348fc8da4302d28b28775]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:48:57.227869 17565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 653fa7149d2e456aaab057aec70ae6f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:57.229743 17916 tablet_bootstrap.cc:492] T 653fa7149d2e456aaab057aec70ae6f6 P fc4631a0d2364673a7e3931e209661dd: Bootstrap starting.
I20260928 22:48:57.230141 17917 tablet_bootstrap.cc:492] T 653fa7149d2e456aaab057aec70ae6f6 P 18bc355d264442c6932effb39d4b3514: Bootstrap starting.
I20260928 22:48:57.230504 17916 tablet_bootstrap.cc:654] T 653fa7149d2e456aaab057aec70ae6f6 P fc4631a0d2364673a7e3931e209661dd: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:57.230806 17917 tablet_bootstrap.cc:654] T 653fa7149d2e456aaab057aec70ae6f6 P 18bc355d264442c6932effb39d4b3514: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:57.231158 17918 tablet_bootstrap.cc:492] T 653fa7149d2e456aaab057aec70ae6f6 P 5558ab0a8f904298adadbae09773156c: Bootstrap starting.
I20260928 22:48:57.231171 17916 log.cc:824] T 653fa7149d2e456aaab057aec70ae6f6 P fc4631a0d2364673a7e3931e209661dd: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:57.231422 17917 log.cc:824] T 653fa7149d2e456aaab057aec70ae6f6 P 18bc355d264442c6932effb39d4b3514: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:57.231892 17918 tablet_bootstrap.cc:654] T 653fa7149d2e456aaab057aec70ae6f6 P 5558ab0a8f904298adadbae09773156c: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:57.231902 17916 tablet_bootstrap.cc:492] T 653fa7149d2e456aaab057aec70ae6f6 P fc4631a0d2364673a7e3931e209661dd: No bootstrap required, opened a new log
I20260928 22:48:57.231990 17916 ts_tablet_manager.cc:1397] T 653fa7149d2e456aaab057aec70ae6f6 P fc4631a0d2364673a7e3931e209661dd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260928 22:48:57.232358 17918 log.cc:824] T 653fa7149d2e456aaab057aec70ae6f6 P 5558ab0a8f904298adadbae09773156c: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:57.233034 17918 tablet_bootstrap.cc:492] T 653fa7149d2e456aaab057aec70ae6f6 P 5558ab0a8f904298adadbae09773156c: No bootstrap required, opened a new log
I20260928 22:48:57.233084 17918 ts_tablet_manager.cc:1397] T 653fa7149d2e456aaab057aec70ae6f6 P 5558ab0a8f904298adadbae09773156c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:48:57.233529 17916 raft_consensus.cc:348] T 653fa7149d2e456aaab057aec70ae6f6 P fc4631a0d2364673a7e3931e209661dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5558ab0a8f904298adadbae09773156c" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 40363 } } peers { permanent_uuid: "18bc355d264442c6932effb39d4b3514" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 37275 } } peers { permanent_uuid: "fc4631a0d2364673a7e3931e209661dd" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 40385 } }
I20260928 22:48:57.233690 17916 raft_consensus.cc:374] T 653fa7149d2e456aaab057aec70ae6f6 P fc4631a0d2364673a7e3931e209661dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:57.233734 17916 raft_consensus.cc:729] T 653fa7149d2e456aaab057aec70ae6f6 P fc4631a0d2364673a7e3931e209661dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc4631a0d2364673a7e3931e209661dd, State: Initialized, Role: FOLLOWER
I20260928 22:48:57.233870 17916 consensus_queue.cc:260] T 653fa7149d2e456aaab057aec70ae6f6 P fc4631a0d2364673a7e3931e209661dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5558ab0a8f904298adadbae09773156c" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 40363 } } peers { permanent_uuid: "18bc355d264442c6932effb39d4b3514" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 37275 } } peers { permanent_uuid: "fc4631a0d2364673a7e3931e209661dd" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 40385 } }
I20260928 22:48:57.234149 17916 ts_tablet_manager.cc:1428] T 653fa7149d2e456aaab057aec70ae6f6 P fc4631a0d2364673a7e3931e209661dd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 22:48:57.234189 17899 heartbeater.cc:503] Master 127.16.121.190:44003 was elected leader, sending a full tablet report...
I20260928 22:48:57.234968 17918 raft_consensus.cc:348] T 653fa7149d2e456aaab057aec70ae6f6 P 5558ab0a8f904298adadbae09773156c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5558ab0a8f904298adadbae09773156c" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 40363 } } peers { permanent_uuid: "18bc355d264442c6932effb39d4b3514" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 37275 } } peers { permanent_uuid: "fc4631a0d2364673a7e3931e209661dd" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 40385 } }
I20260928 22:48:57.235174 17918 raft_consensus.cc:374] T 653fa7149d2e456aaab057aec70ae6f6 P 5558ab0a8f904298adadbae09773156c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:57.235219 17918 raft_consensus.cc:729] T 653fa7149d2e456aaab057aec70ae6f6 P 5558ab0a8f904298adadbae09773156c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5558ab0a8f904298adadbae09773156c, State: Initialized, Role: FOLLOWER
I20260928 22:48:57.235349 17918 consensus_queue.cc:260] T 653fa7149d2e456aaab057aec70ae6f6 P 5558ab0a8f904298adadbae09773156c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5558ab0a8f904298adadbae09773156c" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 40363 } } peers { permanent_uuid: "18bc355d264442c6932effb39d4b3514" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 37275 } } peers { permanent_uuid: "fc4631a0d2364673a7e3931e209661dd" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 40385 } }
I20260928 22:48:57.235541 17917 tablet_bootstrap.cc:492] T 653fa7149d2e456aaab057aec70ae6f6 P 18bc355d264442c6932effb39d4b3514: No bootstrap required, opened a new log
I20260928 22:48:57.235630 17631 heartbeater.cc:503] Master 127.16.121.190:44003 was elected leader, sending a full tablet report...
I20260928 22:48:57.235635 17917 ts_tablet_manager.cc:1397] T 653fa7149d2e456aaab057aec70ae6f6 P 18bc355d264442c6932effb39d4b3514: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260928 22:48:57.235857 17918 ts_tablet_manager.cc:1428] T 653fa7149d2e456aaab057aec70ae6f6 P 5558ab0a8f904298adadbae09773156c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:48:57.237341 17917 raft_consensus.cc:348] T 653fa7149d2e456aaab057aec70ae6f6 P 18bc355d264442c6932effb39d4b3514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5558ab0a8f904298adadbae09773156c" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 40363 } } peers { permanent_uuid: "18bc355d264442c6932effb39d4b3514" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 37275 } } peers { permanent_uuid: "fc4631a0d2364673a7e3931e209661dd" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 40385 } }
I20260928 22:48:57.237511 17917 raft_consensus.cc:374] T 653fa7149d2e456aaab057aec70ae6f6 P 18bc355d264442c6932effb39d4b3514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:57.237560 17917 raft_consensus.cc:729] T 653fa7149d2e456aaab057aec70ae6f6 P 18bc355d264442c6932effb39d4b3514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18bc355d264442c6932effb39d4b3514, State: Initialized, Role: FOLLOWER
I20260928 22:48:57.237670 17917 consensus_queue.cc:260] T 653fa7149d2e456aaab057aec70ae6f6 P 18bc355d264442c6932effb39d4b3514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5558ab0a8f904298adadbae09773156c" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 40363 } } peers { permanent_uuid: "18bc355d264442c6932effb39d4b3514" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 37275 } } peers { permanent_uuid: "fc4631a0d2364673a7e3931e209661dd" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 40385 } }
I20260928 22:48:57.237926 17917 ts_tablet_manager.cc:1428] T 653fa7149d2e456aaab057aec70ae6f6 P 18bc355d264442c6932effb39d4b3514: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 22:48:57.237995 17765 heartbeater.cc:503] Master 127.16.121.190:44003 was elected leader, sending a full tablet report...
W20260928 22:48:57.274312 17766 tablet.cc:2369] T 653fa7149d2e456aaab057aec70ae6f6 P 18bc355d264442c6932effb39d4b3514: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:48:57.367944 17632 tablet.cc:2369] T 653fa7149d2e456aaab057aec70ae6f6 P 5558ab0a8f904298adadbae09773156c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:48:57.400574 17924 raft_consensus.cc:482] T 653fa7149d2e456aaab057aec70ae6f6 P 18bc355d264442c6932effb39d4b3514 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:48:57.400768 17924 raft_consensus.cc:504] T 653fa7149d2e456aaab057aec70ae6f6 P 18bc355d264442c6932effb39d4b3514 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5558ab0a8f904298adadbae09773156c" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 40363 } } peers { permanent_uuid: "18bc355d264442c6932effb39d4b3514" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 37275 } } peers { permanent_uuid: "fc4631a0d2364673a7e3931e209661dd" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 40385 } }
I20260928 22:48:57.401165 17924 leader_election.cc:288] T 653fa7149d2e456aaab057aec70ae6f6 P 18bc355d264442c6932effb39d4b3514 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5558ab0a8f904298adadbae09773156c (127.16.121.129:40363), fc4631a0d2364673a7e3931e209661dd (127.16.121.131:40385)
I20260928 22:48:57.404642 17911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:57.401534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:44995 (local address 127.16.121.129:40363)
0928 22:48:57.401590 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:57.401592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:57.401604 (+ 12us) server_negotiation.cc:407] Connection header received
0928 22:48:57.401638 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:57.401639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:57.401649 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:57.401708 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:48:57.401881 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:57.402433 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:57.403892 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:57.404243 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:57.404308 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:57.404315 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:57.404509 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 22:48:57.404548 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 22:48:57.404649 17767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:57.401405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:40363 (local address 127.16.121.130:44995)
0928 22:48:57.401474 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:57.401488 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:57.401574 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:57.401720 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:57.401722 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:57.401724 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:48:57.401795 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:57.401797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:57.402519 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:57.402520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:57.403800 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:57.403803 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:57.404338 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:57.404339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:57.404431 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:57.404435 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:48:57.404495 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 22:48:57.404565 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":5}
I20260928 22:48:57.404748 17915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:57.401679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:41691 (local address 127.16.121.131:40385)
0928 22:48:57.401741 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:57.401742 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:57.401754 (+ 12us) server_negotiation.cc:407] Connection header received
0928 22:48:57.401832 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:57.401833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:57.401842 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:57.401887 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:48:57.402092 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:57.402572 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:57.403932 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:57.404357 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:57.404433 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:57.404441 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:57.404634 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 22:48:57.404683 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 22:48:57.404994 17925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:57.401558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.131:40385 (local address 127.16.121.130:41691)
0928 22:48:57.401713 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:57.401723 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:57.401765 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:57.401902 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:57.401903 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:57.401905 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:48:57.401977 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:57.401978 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:57.402648 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:57.402649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:57.403800 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:57.403803 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:57.404435 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:57.404436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:57.404536 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:57.404541 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:48:57.404866 (+ 325us) client_negotiation.cc:240] Negotiation successful
0928 22:48:57.404919 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":2,"thread_start_us":103,"threads_started":1}
I20260928 22:48:57.405149 17585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "653fa7149d2e456aaab057aec70ae6f6" candidate_uuid: "18bc355d264442c6932effb39d4b3514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5558ab0a8f904298adadbae09773156c" is_pre_election: true
I20260928 22:48:57.405336 17585 raft_consensus.cc:2454] T 653fa7149d2e456aaab057aec70ae6f6 P 5558ab0a8f904298adadbae09773156c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18bc355d264442c6932effb39d4b3514 in term 0.
I20260928 22:48:57.405390 17853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "653fa7149d2e456aaab057aec70ae6f6" candidate_uuid: "18bc355d264442c6932effb39d4b3514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc4631a0d2364673a7e3931e209661dd" is_pre_election: true
I20260928 22:48:57.405601 17853 raft_consensus.cc:2454] T 653fa7149d2e456aaab057aec70ae6f6 P fc4631a0d2364673a7e3931e209661dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18bc355d264442c6932effb39d4b3514 in term 0.
I20260928 22:48:57.405602 17653 leader_election.cc:302] T 653fa7149d2e456aaab057aec70ae6f6 P 18bc355d264442c6932effb39d4b3514 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18bc355d264442c6932effb39d4b3514, 5558ab0a8f904298adadbae09773156c; no voters:
I20260928 22:48:57.405726 17924 raft_consensus.cc:2781] T 653fa7149d2e456aaab057aec70ae6f6 P 18bc355d264442c6932effb39d4b3514 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:48:57.405781 17924 raft_consensus.cc:482] T 653fa7149d2e456aaab057aec70ae6f6 P 18bc355d264442c6932effb39d4b3514 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:48:57.405803 17924 raft_consensus.cc:3037] T 653fa7149d2e456aaab057aec70ae6f6 P 18bc355d264442c6932effb39d4b3514 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:57.406589 17924 raft_consensus.cc:504] T 653fa7149d2e456aaab057aec70ae6f6 P 18bc355d264442c6932effb39d4b3514 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5558ab0a8f904298adadbae09773156c" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 40363 } } peers { permanent_uuid: "18bc355d264442c6932effb39d4b3514" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 37275 } } peers { permanent_uuid: "fc4631a0d2364673a7e3931e209661dd" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 40385 } }
I20260928 22:48:57.406719 17924 leader_election.cc:288] T 653fa7149d2e456aaab057aec70ae6f6 P 18bc355d264442c6932effb39d4b3514 [CANDIDATE]: Term 1 election: Requested vote from peers 5558ab0a8f904298adadbae09773156c (127.16.121.129:40363), fc4631a0d2364673a7e3931e209661dd (127.16.121.131:40385)
I20260928 22:48:57.406889 17585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "653fa7149d2e456aaab057aec70ae6f6" candidate_uuid: "18bc355d264442c6932effb39d4b3514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5558ab0a8f904298adadbae09773156c"
I20260928 22:48:57.406955 17853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "653fa7149d2e456aaab057aec70ae6f6" candidate_uuid: "18bc355d264442c6932effb39d4b3514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc4631a0d2364673a7e3931e209661dd"
I20260928 22:48:57.406989 17585 raft_consensus.cc:3037] T 653fa7149d2e456aaab057aec70ae6f6 P 5558ab0a8f904298adadbae09773156c [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:57.407024 17853 raft_consensus.cc:3037] T 653fa7149d2e456aaab057aec70ae6f6 P fc4631a0d2364673a7e3931e209661dd [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:57.407823 17853 raft_consensus.cc:2454] T 653fa7149d2e456aaab057aec70ae6f6 P fc4631a0d2364673a7e3931e209661dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18bc355d264442c6932effb39d4b3514 in term 1.
I20260928 22:48:57.407912 17585 raft_consensus.cc:2454] T 653fa7149d2e456aaab057aec70ae6f6 P 5558ab0a8f904298adadbae09773156c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18bc355d264442c6932effb39d4b3514 in term 1.
I20260928 22:48:57.407986 17653 leader_election.cc:302] T 653fa7149d2e456aaab057aec70ae6f6 P 18bc355d264442c6932effb39d4b3514 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18bc355d264442c6932effb39d4b3514, fc4631a0d2364673a7e3931e209661dd; no voters:
I20260928 22:48:57.408111 17924 raft_consensus.cc:2781] T 653fa7149d2e456aaab057aec70ae6f6 P 18bc355d264442c6932effb39d4b3514 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:48:57.408258 17924 raft_consensus.cc:686] T 653fa7149d2e456aaab057aec70ae6f6 P 18bc355d264442c6932effb39d4b3514 [term 1 LEADER]: Becoming Leader. State: Replica: 18bc355d264442c6932effb39d4b3514, State: Running, Role: LEADER
I20260928 22:48:57.408351 17924 consensus_queue.cc:237] T 653fa7149d2e456aaab057aec70ae6f6 P 18bc355d264442c6932effb39d4b3514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5558ab0a8f904298adadbae09773156c" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 40363 } } peers { permanent_uuid: "18bc355d264442c6932effb39d4b3514" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 37275 } } peers { permanent_uuid: "fc4631a0d2364673a7e3931e209661dd" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 40385 } }
I20260928 22:48:57.409197 17439 catalog_manager.cc:5495] T 653fa7149d2e456aaab057aec70ae6f6 P 18bc355d264442c6932effb39d4b3514 reported cstate change: term changed from 0 to 1, leader changed from <none> to 18bc355d264442c6932effb39d4b3514 (127.16.121.130). New cstate: current_term: 1 leader_uuid: "18bc355d264442c6932effb39d4b3514" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5558ab0a8f904298adadbae09773156c" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 40363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18bc355d264442c6932effb39d4b3514" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 37275 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc4631a0d2364673a7e3931e209661dd" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 40385 } health_report { overall_health: UNKNOWN } } }
W20260928 22:48:57.441239 17900 tablet.cc:2369] T 653fa7149d2e456aaab057aec70ae6f6 P fc4631a0d2364673a7e3931e209661dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 22:48:57 dist-test-slave-56vw krb5kdc[17395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.121.129@KRBTEST.COM
I20260928 22:48:57.458968 17911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:57.450942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42324 (local address 127.16.121.129:40363)
0928 22:48:57.451029 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:57.451031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:57.451156 (+ 125us) server_negotiation.cc:407] Connection header received
0928 22:48:57.451199 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:57.451200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:57.451211 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:57.451270 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:57.451867 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:57.452434 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:57.453193 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:57.453394 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:57.453468 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:57.455946 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:57.455966 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:57.455967 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:57.455975 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:57.457579 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:57.457985 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:57.457988 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:57.457989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:57.458051 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:57.458261 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:57.458262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:57.458262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:57.458527 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:57.458733 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:57.458773 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 22:48:57.458863 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Sep 28 22:48:57 dist-test-slave-56vw krb5kdc[17395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635737, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 22:48:57 dist-test-slave-56vw krb5kdc[17395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635737, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.121.129@KRBTEST.COM
I20260928 22:48:57.483414 17911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:57.475403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42328 (local address 127.16.121.129:40363)
0928 22:48:57.475480 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:57.475482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:57.475613 (+ 131us) server_negotiation.cc:407] Connection header received
0928 22:48:57.475682 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:57.475683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:57.475708 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:57.475775 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:57.476419 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:57.476924 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:57.477790 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:57.477994 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:57.478055 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:57.480566 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:57.480567 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:57.480567 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:57.480575 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:57.482061 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:57.482534 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:57.482535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:57.482536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:57.482593 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:57.482776 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:57.482777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:57.482777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:57.482882 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:57.482975 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:57.483278 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 22:48:57.483345 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 22:48:57 dist-test-slave-56vw krb5kdc[17395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 22:48:57 dist-test-slave-56vw krb5kdc[17395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.121.129@KRBTEST.COM
I20260928 22:48:57.509796 17911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:57.502026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42344 (local address 127.16.121.129:40363)
0928 22:48:57.502091 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:57.502093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:57.502127 (+ 34us) server_negotiation.cc:407] Connection header received
0928 22:48:57.502195 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:57.502196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:57.502205 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:57.502257 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:57.502839 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:57.503402 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:57.504162 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:57.504361 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:57.504425 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:57.507311 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:57.507312 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:57.507313 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:57.507320 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:57.508793 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:57.509175 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:57.509176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:57.509176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:57.509227 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:57.509396 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:57.509397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:57.509397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:57.509489 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:57.509557 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:57.509650 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 22:48:57.509706 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260928 22:48:57.510288 17545 tablet_service.cc:2908] Rejecting scan request for tablet 653fa7149d2e456aaab057aec70ae6f6: Uninitialized: clean time has not yet been initialized
I20260928 22:48:57.511296 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 17502
I20260928 22:48:57.516906 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 17636
I20260928 22:48:57.522082 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 17770
I20260928 22:48:57.527247 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 17408
2026-09-28T22:48:57Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2231 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:48:57 dist-test-slave-56vw krb5kdc[17940](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:48:57 dist-test-slave-56vw krb5kdc[17940](info): set up 2 sockets
Sep 28 22:48:57 dist-test-slave-56vw krb5kdc[17940](info): commencing operation
krb5kdc: starting...
W20260928 22:48:58.590720 16870 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Sep 28 22:48:58 dist-test-slave-56vw krb5kdc[17940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:48:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:48:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.121.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
I20260928 22:48:58.715137 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:33833
--webserver_interface=127.16.121.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:37303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:33833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:48:58.800218 17953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:58.800477 17953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:58.800516 17953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:58.801751 17953 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:48:58.801826 17953 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 22:48:58.801846 17953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:58.801868 17953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:58.801900 17953 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:48:58.801929 17953 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:48:58.803524 17953 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:37303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:33833
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:33833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.121.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:58.803815 17953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:58.804054 17953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:58.809441 17958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:58.809456 17959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:58.809455 17961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:58.809636 17953 server_base.cc:1034] running on GCE node
I20260928 22:48:58.810010 17953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:58.810284 17953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:58.811478 17953 hybrid_clock.cc:648] HybridClock initialized: now 1790635738811441 us; error 61 us; skew 500 ppm
Sep 28 22:48:58 dist-test-slave-56vw krb5kdc[17940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635738, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:48:58.821587 17953 init.cc:377] Logged in from keytab as kudu/127.16.121.190@KRBTEST.COM (short username kudu)
I20260928 22:48:58.822091 17953 webserver.cc:460] Webserver started at http://127.16.121.190:46873/ using document root <none> and password file <none>
I20260928 22:48:58.822342 17953 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:58.822412 17953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:58.822566 17953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:58.824090 17953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "545769a425e048d9a2cb5f05d46f2f68"
format_stamp: "Formatted at 2026-09-28 22:48:58 on dist-test-slave-56vw"
server_key: "76cd50c6c38dac0106328bfec7ed5df8"
server_key_iv: "3be6b97aa24a6250a6a5f5e94ae1a150"
server_key_version: "encryptionkey@0"
I20260928 22:48:58.824542 17953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "545769a425e048d9a2cb5f05d46f2f68"
format_stamp: "Formatted at 2026-09-28 22:48:58 on dist-test-slave-56vw"
server_key: "76cd50c6c38dac0106328bfec7ed5df8"
server_key_iv: "3be6b97aa24a6250a6a5f5e94ae1a150"
server_key_version: "encryptionkey@0"
I20260928 22:48:58.825956 17953 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:48:58.826763 17968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:58.826936 17953 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:48:58.827080 17953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "545769a425e048d9a2cb5f05d46f2f68"
format_stamp: "Formatted at 2026-09-28 22:48:58 on dist-test-slave-56vw"
server_key: "76cd50c6c38dac0106328bfec7ed5df8"
server_key_iv: "3be6b97aa24a6250a6a5f5e94ae1a150"
server_key_version: "encryptionkey@0"
I20260928 22:48:58.827165 17953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:58.839394 17953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:58.841836 17953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:58.842017 17953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:58.846836 17953 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.190:33833
I20260928 22:48:58.846916 18030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.190:33833 every 8 connection(s)
I20260928 22:48:58.847271 17953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:48:58.847916 18031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:58.850313 18031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 545769a425e048d9a2cb5f05d46f2f68: Bootstrap starting.
I20260928 22:48:58.850570 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 17953
I20260928 22:48:58.850665 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:48:58.850818 16870 external_mini_cluster.cc:1428] Setting key 5ce77aece9a7862b2c18a1d4edc777d2
I20260928 22:48:58.851007 18031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 545769a425e048d9a2cb5f05d46f2f68: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:58.851363 18031 log.cc:824] T 00000000000000000000000000000000 P 545769a425e048d9a2cb5f05d46f2f68: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:58.852082 18031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 545769a425e048d9a2cb5f05d46f2f68: No bootstrap required, opened a new log
I20260928 22:48:58.853466 18031 raft_consensus.cc:348] T 00000000000000000000000000000000 P 545769a425e048d9a2cb5f05d46f2f68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "545769a425e048d9a2cb5f05d46f2f68" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 33833 } }
I20260928 22:48:58.853597 18031 raft_consensus.cc:374] T 00000000000000000000000000000000 P 545769a425e048d9a2cb5f05d46f2f68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:58.853619 18031 raft_consensus.cc:729] T 00000000000000000000000000000000 P 545769a425e048d9a2cb5f05d46f2f68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 545769a425e048d9a2cb5f05d46f2f68, State: Initialized, Role: FOLLOWER
I20260928 22:48:58.853750 18031 consensus_queue.cc:260] T 00000000000000000000000000000000 P 545769a425e048d9a2cb5f05d46f2f68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "545769a425e048d9a2cb5f05d46f2f68" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 33833 } }
I20260928 22:48:58.853818 18031 raft_consensus.cc:388] T 00000000000000000000000000000000 P 545769a425e048d9a2cb5f05d46f2f68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:48:58.853869 18031 raft_consensus.cc:482] T 00000000000000000000000000000000 P 545769a425e048d9a2cb5f05d46f2f68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:48:58.853922 18031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 545769a425e048d9a2cb5f05d46f2f68 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:58.854475 18031 raft_consensus.cc:504] T 00000000000000000000000000000000 P 545769a425e048d9a2cb5f05d46f2f68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "545769a425e048d9a2cb5f05d46f2f68" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 33833 } }
I20260928 22:48:58.854592 18031 leader_election.cc:302] T 00000000000000000000000000000000 P 545769a425e048d9a2cb5f05d46f2f68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 545769a425e048d9a2cb5f05d46f2f68; no voters:
I20260928 22:48:58.854779 18031 leader_election.cc:288] T 00000000000000000000000000000000 P 545769a425e048d9a2cb5f05d46f2f68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:48:58.854804 18036 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 545769a425e048d9a2cb5f05d46f2f68 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:48:58.854925 18036 raft_consensus.cc:686] T 00000000000000000000000000000000 P 545769a425e048d9a2cb5f05d46f2f68 [term 1 LEADER]: Becoming Leader. State: Replica: 545769a425e048d9a2cb5f05d46f2f68, State: Running, Role: LEADER
I20260928 22:48:58.855005 18036 consensus_queue.cc:237] T 00000000000000000000000000000000 P 545769a425e048d9a2cb5f05d46f2f68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "545769a425e048d9a2cb5f05d46f2f68" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 33833 } }
I20260928 22:48:58.855077 18031 sys_catalog.cc:564] T 00000000000000000000000000000000 P 545769a425e048d9a2cb5f05d46f2f68 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:48:58.855355 18036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 545769a425e048d9a2cb5f05d46f2f68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 545769a425e048d9a2cb5f05d46f2f68. Latest consensus state: current_term: 1 leader_uuid: "545769a425e048d9a2cb5f05d46f2f68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "545769a425e048d9a2cb5f05d46f2f68" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 33833 } } }
I20260928 22:48:58.855444 18036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 545769a425e048d9a2cb5f05d46f2f68 [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:58.855369 18037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 545769a425e048d9a2cb5f05d46f2f68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "545769a425e048d9a2cb5f05d46f2f68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "545769a425e048d9a2cb5f05d46f2f68" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 33833 } } }
I20260928 22:48:58.855731 18037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 545769a425e048d9a2cb5f05d46f2f68 [sys.catalog]: This master's current role is: LEADER
I20260928 22:48:58.856110 18041 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:48:58.856954 18041 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 22:48:58 dist-test-slave-56vw krb5kdc[17940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:48:58.859108 18041 catalog_manager.cc:1347] Generated new cluster ID: 96e259eaa8b24542b2977c456fb92588
I20260928 22:48:58.859483 18041 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:48:58.866609 18034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:58.851364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40744 (local address 127.16.121.190:33833)
0928 22:48:58.851613 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:58.851624 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:58.851656 (+ 32us) server_negotiation.cc:407] Connection header received
0928 22:48:58.852100 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:58.852106 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:58.852187 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:58.852352 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:58.852950 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:58.854333 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:58.855333 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:58.855975 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:58.856094 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:58.859016 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:58.859020 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:58.859020 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:58.859034 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:58.864852 (+ 5818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:58.865319 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:58.865322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:58.865322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:58.865440 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:58.865634 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:58.865635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:58.865635 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:58.865978 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:58.866140 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:58.866222 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 22:48:58.866370 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":77,"threads_started":1}
I20260928 22:48:58.882169 18041 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:48:58.882813 18041 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:48:58.890515 18041 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 545769a425e048d9a2cb5f05d46f2f68: Generated new TSK 0
I20260928 22:48:58.890733 18041 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.121.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab.
Entry for principal kudu/127.16.121.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab.
I20260928 22:48:58.922130 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.129:0
--local_ip_for_outbound_sockets=127.16.121.129
--webserver_interface=127.16.121.129
--webserver_port=0
--tserver_master_addrs=127.16.121.190:33833
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab
--principal=kudu/127.16.121.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:37303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:48:59.006103 18057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:59.006319 18057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:59.006345 18057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:59.007663 18057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:59.007712 18057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:59.007735 18057 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 22:48:59.007758 18057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.129
I20260928 22:48:59.009267 18057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:37303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab
--principal=kudu/127.16.121.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.121.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.121.190:33833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.129
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:59.009523 18057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:59.009759 18057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:59.015229 18065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:59.015211 18062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:59.015206 18063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:59.015246 18057 server_base.cc:1034] running on GCE node
I20260928 22:48:59.015569 18057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:59.015810 18057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:59.016991 18057 hybrid_clock.cc:648] HybridClock initialized: now 1790635739016970 us; error 32 us; skew 500 ppm
Sep 28 22:48:59 dist-test-slave-56vw krb5kdc[17940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635739, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:48:59.019790 18057 init.cc:377] Logged in from keytab as kudu/127.16.121.129@KRBTEST.COM (short username kudu)
I20260928 22:48:59.020241 18057 webserver.cc:460] Webserver started at http://127.16.121.129:38889/ using document root <none> and password file <none>
I20260928 22:48:59.020533 18057 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:59.020607 18057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:59.020746 18057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:59.021822 18057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-0/data/instance:
uuid: "c19bcd4777c748638a4a12129d17face"
format_stamp: "Formatted at 2026-09-28 22:48:59 on dist-test-slave-56vw"
server_key: "ad93831f1a0de7826d8b07b9b0e423a3"
server_key_iv: "9e249044c15952a376ffaee008d1f136"
server_key_version: "encryptionkey@0"
I20260928 22:48:59.022193 18057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance:
uuid: "c19bcd4777c748638a4a12129d17face"
format_stamp: "Formatted at 2026-09-28 22:48:59 on dist-test-slave-56vw"
server_key: "ad93831f1a0de7826d8b07b9b0e423a3"
server_key_iv: "9e249044c15952a376ffaee008d1f136"
server_key_version: "encryptionkey@0"
I20260928 22:48:59.023617 18057 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:48:59.024302 18072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:59.024490 18057 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:48:59.024603 18057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-0/wal
uuid: "c19bcd4777c748638a4a12129d17face"
format_stamp: "Formatted at 2026-09-28 22:48:59 on dist-test-slave-56vw"
server_key: "ad93831f1a0de7826d8b07b9b0e423a3"
server_key_iv: "9e249044c15952a376ffaee008d1f136"
server_key_version: "encryptionkey@0"
I20260928 22:48:59.024675 18057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:59.052325 18057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:59.054448 18057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:59.054657 18057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:59.055325 18057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:48:59.055382 18057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:59.055426 18057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:48:59.055521 18057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 22:48:59 dist-test-slave-56vw krb5kdc[17940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635739, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:48:59.062790 18057 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.129:41783
I20260928 22:48:59.062867 18187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.129:41783 every 8 connection(s)
I20260928 22:48:59.063239 18057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
I20260928 22:48:59.066371 18034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:59.055856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:56387 (local address 127.16.121.190:33833)
0928 22:48:59.055924 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:59.055927 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:59.056222 (+ 295us) server_negotiation.cc:407] Connection header received
0928 22:48:59.056774 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:59.056775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:59.056787 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:59.056844 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:59.057624 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.058148 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.059285 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.059562 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.059641 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:59.063219 (+ 3578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:59.063221 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:59.063221 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:59.063236 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:59.064941 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:59.065377 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:59.065379 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:59.065379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:59.065439 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:59.065694 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:59.065695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:59.065695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:59.065822 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:59.065892 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:59.066174 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 22:48:59.066274 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 22:48:59.066581 18081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:59.055858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:33833 (local address 127.16.121.129:56387)
0928 22:48:59.056077 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:59.056134 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:59.056652 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:59.056928 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:59.056933 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:59.057288 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:59.057528 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:59.057533 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.058241 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.058242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:59.059201 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:59.059204 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.059648 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.059649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:59.059781 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:59.060350 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:48:59.060363 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:59.063106 (+ 2743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:59.065054 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:59.065061 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:59.065063 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:59.065293 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:59.065506 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:59.065507 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:59.065507 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:59.065622 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:59.065894 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:59.065901 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:59.066048 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 22:48:59.066157 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 22:48:59.066326 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":115,"threads_started":1}
I20260928 22:48:59.067126 18188 heartbeater.cc:347] Connected to a master server at 127.16.121.190:33833
I20260928 22:48:59.067241 18188 heartbeater.cc:464] Registering TS with master...
I20260928 22:48:59.067457 18188 heartbeater.cc:511] Master 127.16.121.190:33833 requested a full tablet report, sending...
I20260928 22:48:59.067539 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 18057
I20260928 22:48:59.067642 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance
I20260928 22:48:59.067824 16870 external_mini_cluster.cc:1428] Setting key 87b9a9353027cda847a12d939ace0989
I20260928 22:48:59.068048 17984 ts_manager.cc:194] Registered new tserver with Master: c19bcd4777c748638a4a12129d17face (127.16.121.129:41783)
I20260928 22:48:59.068755 17984 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.129@KRBTEST.COM'} at 127.16.121.129:56387
WARNING: no policy specified for kudu/127.16.121.130@KRBTEST.COM; defaulting to no policy
I20260928 22:48:59.075398 18034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:59.068203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:60139 (local address 127.16.121.190:33833)
0928 22:48:59.068238 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:59.068240 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:59.068461 (+ 221us) server_negotiation.cc:407] Connection header received
0928 22:48:59.069987 (+ 1526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:59.069988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:59.069999 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:59.070063 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:59.070375 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.070856 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.071645 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.071844 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.071879 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:59.072927 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:59.072928 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:59.072929 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:59.072936 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:59.074256 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:59.074604 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:59.074606 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:59.074606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:59.074660 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:59.074858 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:59.074858 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:59.074859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:59.074968 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:59.075071 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:59.075224 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 22:48:59.075302 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 22:48:59.075403 18194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:59.068209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:33833 (local address 127.16.121.129:60139)
0928 22:48:59.068400 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:59.068412 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:59.069904 (+ 1492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:59.070068 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:59.070069 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:59.070250 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:59.070300 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:59.070302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.070949 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.070950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:59.071529 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:59.071531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.071941 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.071941 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:59.071996 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:59.072504 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:48:59.072508 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:59.072837 (+ 329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:59.074356 (+ 1519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:59.074357 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:59.074358 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:59.074534 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:59.074728 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:59.074729 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:59.074730 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:59.074787 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:59.075070 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:59.075070 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:59.075116 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 22:48:59.075199 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 22:48:59.075302 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab.
Entry for principal kudu/127.16.121.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab.
I20260928 22:48:59.094310 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.130:0
--local_ip_for_outbound_sockets=127.16.121.130
--webserver_interface=127.16.121.130
--webserver_port=0
--tserver_master_addrs=127.16.121.190:33833
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab
--principal=kudu/127.16.121.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:37303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:48:59.179737 18197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:59.179991 18197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:59.180032 18197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:59.181281 18197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:59.181351 18197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:59.181396 18197 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 22:48:59.181433 18197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.130
I20260928 22:48:59.182955 18197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:37303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab
--principal=kudu/127.16.121.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.121.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.121.190:33833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.130
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:59.183286 18197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:59.183530 18197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:59.188737 18202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:59.188753 18203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:59.188753 18205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:59.189095 18197 server_base.cc:1034] running on GCE node
I20260928 22:48:59.189366 18197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:59.189625 18197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:59.190783 18197 hybrid_clock.cc:648] HybridClock initialized: now 1790635739190778 us; error 37 us; skew 500 ppm
Sep 28 22:48:59 dist-test-slave-56vw krb5kdc[17940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635739, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:48:59.193428 18197 init.cc:377] Logged in from keytab as kudu/127.16.121.130@KRBTEST.COM (short username kudu)
I20260928 22:48:59.193908 18197 webserver.cc:460] Webserver started at http://127.16.121.130:33641/ using document root <none> and password file <none>
I20260928 22:48:59.194156 18197 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:59.194222 18197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:59.194360 18197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:59.195521 18197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-1/data/instance:
uuid: "f168b2278558423e9b6d11ecc9314584"
format_stamp: "Formatted at 2026-09-28 22:48:59 on dist-test-slave-56vw"
server_key: "6e3fa411add274fccfa81f1221576639"
server_key_iv: "5b2e1f69ccbdad872ab17616b7d8323c"
server_key_version: "encryptionkey@0"
I20260928 22:48:59.195883 18197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance:
uuid: "f168b2278558423e9b6d11ecc9314584"
format_stamp: "Formatted at 2026-09-28 22:48:59 on dist-test-slave-56vw"
server_key: "6e3fa411add274fccfa81f1221576639"
server_key_iv: "5b2e1f69ccbdad872ab17616b7d8323c"
server_key_version: "encryptionkey@0"
I20260928 22:48:59.197227 18197 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:48:59.197970 18212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:59.198143 18197 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:48:59.198243 18197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-1/wal
uuid: "f168b2278558423e9b6d11ecc9314584"
format_stamp: "Formatted at 2026-09-28 22:48:59 on dist-test-slave-56vw"
server_key: "6e3fa411add274fccfa81f1221576639"
server_key_iv: "5b2e1f69ccbdad872ab17616b7d8323c"
server_key_version: "encryptionkey@0"
I20260928 22:48:59.198314 18197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:59.219619 18197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:59.222154 18197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:59.222294 18197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:59.223094 18197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:48:59.223158 18197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:59.223193 18197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:48:59.223238 18197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 22:48:59 dist-test-slave-56vw krb5kdc[17940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635739, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.130@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:48:59.230991 18197 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.130:32883
I20260928 22:48:59.231249 18327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.130:32883 every 8 connection(s)
I20260928 22:48:59.231495 18197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
I20260928 22:48:59.234670 18034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:59.223592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:54885 (local address 127.16.121.190:33833)
0928 22:48:59.223664 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:59.223667 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:59.223900 (+ 233us) server_negotiation.cc:407] Connection header received
0928 22:48:59.224412 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:59.224414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:59.224431 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:59.224534 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:59.225445 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.226402 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.228168 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.228380 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.228476 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:59.231630 (+ 3154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:59.231632 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:59.231633 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:59.231643 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:59.233221 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:59.233664 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:59.233666 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:59.233667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:59.233734 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:59.233994 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:59.233995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:59.233996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:59.234125 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:59.234199 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:59.234489 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 22:48:59.234573 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 22:48:59.234884 18221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:59.223582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:33833 (local address 127.16.121.130:54885)
0928 22:48:59.223752 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:59.223795 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:59.224306 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:59.224638 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:59.224643 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:59.224999 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:59.225327 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:59.225334 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.226546 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.226547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:59.228046 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:59.228051 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.228501 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.228502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:59.228665 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:59.229340 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:48:59.229354 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:59.231491 (+ 2137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:59.233330 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:59.233337 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:59.233340 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:59.233584 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:59.233800 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:59.233801 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:59.233802 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:59.233927 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:59.234188 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:59.234194 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:59.234343 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 22:48:59.234467 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 22:48:59.234641 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":95,"threads_started":1}
I20260928 22:48:59.235440 18328 heartbeater.cc:347] Connected to a master server at 127.16.121.190:33833
I20260928 22:48:59.235559 18328 heartbeater.cc:464] Registering TS with master...
I20260928 22:48:59.235746 18328 heartbeater.cc:511] Master 127.16.121.190:33833 requested a full tablet report, sending...
I20260928 22:48:59.236109 17985 ts_manager.cc:194] Registered new tserver with Master: f168b2278558423e9b6d11ecc9314584 (127.16.121.130:32883)
I20260928 22:48:59.236532 17985 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.130@KRBTEST.COM'} at 127.16.121.130:54885
I20260928 22:48:59.239711 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 18197
I20260928 22:48:59.239837 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance
I20260928 22:48:59.240016 16870 external_mini_cluster.cc:1428] Setting key 44158e3b87f85ed6e58235380b7d4c13
I20260928 22:48:59.242522 18034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:59.236589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:58333 (local address 127.16.121.190:33833)
0928 22:48:59.236629 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:59.236631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:59.236739 (+ 108us) server_negotiation.cc:407] Connection header received
0928 22:48:59.236810 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:59.236810 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:59.236821 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:59.236862 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:59.237223 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.237703 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.238406 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.238611 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.238684 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:59.239753 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:59.239754 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:59.239754 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:59.239762 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:59.241195 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:59.241560 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:59.241561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:59.241561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:59.241618 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:59.241840 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:59.241841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:59.241841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:59.242061 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:59.242112 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:59.242344 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 22:48:59.242422 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 22:48:59.242575 18334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:59.236509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:33833 (local address 127.16.121.130:58333)
0928 22:48:59.236689 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:59.236700 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:59.236753 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:59.236887 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:59.236888 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:59.237059 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:59.237146 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:59.237148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.237793 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.237794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:59.238325 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:59.238327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.238682 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.238683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:59.238766 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:59.239247 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:48:59.239252 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:59.239660 (+ 408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:59.241296 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:59.241297 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:59.241298 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:59.241479 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:59.241681 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:59.241682 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:59.241682 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:59.241740 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:59.242166 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:59.242167 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:59.242220 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 22:48:59.242295 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 22:48:59.242442 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":3,"thread_start_us":72,"threads_started":1}
WARNING: no policy specified for kudu/127.16.121.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab.
Entry for principal kudu/127.16.121.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab.
I20260928 22:48:59.270125 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.131:0
--local_ip_for_outbound_sockets=127.16.121.131
--webserver_interface=127.16.121.131
--webserver_port=0
--tserver_master_addrs=127.16.121.190:33833
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab
--principal=kudu/127.16.121.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:37303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:48:59.355100 18337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:48:59.355384 18337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:48:59.355433 18337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:48:59.356814 18337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:48:59.356896 18337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:48:59.356931 18337 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 22:48:59.356967 18337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.131
I20260928 22:48:59.358598 18337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:37303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab
--principal=kudu/127.16.121.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.121.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.121.190:33833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.131
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:48:59.358800 18337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:48:59.358981 18337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:48:59.364483 18342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:59.364488 18345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:48:59.364483 18343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:48:59.364660 18337 server_base.cc:1034] running on GCE node
I20260928 22:48:59.365067 18337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:48:59.365335 18337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:48:59.366510 18337 hybrid_clock.cc:648] HybridClock initialized: now 1790635739366509 us; error 54 us; skew 500 ppm
Sep 28 22:48:59 dist-test-slave-56vw krb5kdc[17940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635739, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:48:59.369380 18337 init.cc:377] Logged in from keytab as kudu/127.16.121.131@KRBTEST.COM (short username kudu)
I20260928 22:48:59.369829 18337 webserver.cc:460] Webserver started at http://127.16.121.131:36317/ using document root <none> and password file <none>
I20260928 22:48:59.370057 18337 fs_manager.cc:359] Metadata directory not provided
I20260928 22:48:59.370105 18337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:48:59.370230 18337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:48:59.371390 18337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-2/data/instance:
uuid: "d5b10d16e38945e287215c52aa4c9aca"
format_stamp: "Formatted at 2026-09-28 22:48:59 on dist-test-slave-56vw"
server_key: "d1515939fdd6bea96bfd12810562eb74"
server_key_iv: "82710a3d2c69bd8ca3ea3a548f9b8a4d"
server_key_version: "encryptionkey@0"
I20260928 22:48:59.371750 18337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance:
uuid: "d5b10d16e38945e287215c52aa4c9aca"
format_stamp: "Formatted at 2026-09-28 22:48:59 on dist-test-slave-56vw"
server_key: "d1515939fdd6bea96bfd12810562eb74"
server_key_iv: "82710a3d2c69bd8ca3ea3a548f9b8a4d"
server_key_version: "encryptionkey@0"
I20260928 22:48:59.373117 18337 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 22:48:59.373874 18352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:59.374186 18337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 22:48:59.374276 18337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-2/wal
uuid: "d5b10d16e38945e287215c52aa4c9aca"
format_stamp: "Formatted at 2026-09-28 22:48:59 on dist-test-slave-56vw"
server_key: "d1515939fdd6bea96bfd12810562eb74"
server_key_iv: "82710a3d2c69bd8ca3ea3a548f9b8a4d"
server_key_version: "encryptionkey@0"
I20260928 22:48:59.374348 18337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:48:59.391402 18337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:48:59.393770 18337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:48:59.393936 18337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:48:59.394557 18337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:48:59.394623 18337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:59.394670 18337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:48:59.394698 18337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 22:48:59 dist-test-slave-56vw krb5kdc[17940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635739, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.131@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:48:59.401926 18337 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.131:34453
I20260928 22:48:59.402052 18467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.131:34453 every 8 connection(s)
I20260928 22:48:59.402390 18337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
I20260928 22:48:59.405105 18034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:59.394972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:46227 (local address 127.16.121.190:33833)
0928 22:48:59.395040 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:59.395065 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:59.395318 (+ 253us) server_negotiation.cc:407] Connection header received
0928 22:48:59.395852 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:59.395854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:59.395868 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:59.395959 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:59.396702 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.397547 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.399172 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.399387 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.399473 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:59.402117 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:59.402119 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:59.402119 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:59.402130 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:59.403674 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:59.404090 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:59.404091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:59.404091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:59.404152 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:59.404470 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:59.404471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:59.404472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:59.404645 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:59.404713 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:59.404964 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 22:48:59.405036 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 22:48:59.405270 18361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:59.394983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:33833 (local address 127.16.121.131:46227)
0928 22:48:59.395172 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:59.395216 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:59.395736 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:59.396017 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:59.396022 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:59.396271 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:59.396585 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:59.396603 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.397640 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.397641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:59.399077 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:59.399081 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.399499 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.399500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:59.399642 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:59.400286 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:48:59.400295 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:59.402007 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:59.403756 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:59.403761 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:59.403763 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:59.403992 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:59.404244 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:59.404245 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:59.404245 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:59.404360 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:59.404715 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:59.404720 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:59.404839 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 22:48:59.404936 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 22:48:59.405066 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":116,"threads_started":1}
I20260928 22:48:59.405704 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 18337
I20260928 22:48:59.405795 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance
I20260928 22:48:59.405992 16870 external_mini_cluster.cc:1428] Setting key fb7b7313d7fc948341d738ab2f48c15e
I20260928 22:48:59.406862 18468 heartbeater.cc:347] Connected to a master server at 127.16.121.190:33833
I20260928 22:48:59.406955 18468 heartbeater.cc:464] Registering TS with master...
I20260928 22:48:59.407173 18468 heartbeater.cc:511] Master 127.16.121.190:33833 requested a full tablet report, sending...
I20260928 22:48:59.407495 17985 ts_manager.cc:194] Registered new tserver with Master: d5b10d16e38945e287215c52aa4c9aca (127.16.121.131:34453)
I20260928 22:48:59.407889 17985 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.131@KRBTEST.COM'} at 127.16.121.131:46227
I20260928 22:48:59.412834 18034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:59.406993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:58955 (local address 127.16.121.190:33833)
0928 22:48:59.407072 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:59.407074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:59.407307 (+ 233us) server_negotiation.cc:407] Connection header received
0928 22:48:59.407381 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:59.407382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:59.407394 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:59.407440 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:59.407806 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.408344 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.409041 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.409246 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.409328 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:59.410293 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:59.410294 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:59.410294 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:59.410301 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:59.411713 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:59.412068 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:59.412069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:59.412069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:59.412127 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:59.412316 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:59.412317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:59.412317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:59.412435 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:59.412502 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:59.412660 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 22:48:59.412738 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:48:59.412847 18474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:59.407104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:33833 (local address 127.16.121.131:58955)
0928 22:48:59.407248 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:59.407261 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:59.407322 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:59.407453 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:59.407454 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:59.407681 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:59.407736 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:59.407737 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.408442 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.408443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:59.408961 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:59.408963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.409322 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.409323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:59.409411 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:59.409886 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:48:59.409891 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:59.410188 (+ 297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:59.411811 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:59.411812 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:59.411812 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:59.411985 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:59.412189 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:59.412190 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:59.412190 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:59.412249 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:59.412508 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:59.412509 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:59.412556 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 22:48:59.412636 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 22:48:59.412741 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":72,"threads_started":1}
I20260928 22:48:59.417821 16870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 22:48:59 dist-test-slave-56vw krb5kdc[17940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635739, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 22:48:59 dist-test-slave-56vw krb5kdc[17940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635739, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:48:59.441576 18034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:59.433935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40750 (local address 127.16.121.190:33833)
0928 22:48:59.433998 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:59.434000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:59.434025 (+ 25us) server_negotiation.cc:407] Connection header received
0928 22:48:59.434118 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:59.434118 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:59.434128 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:59.434184 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:59.434896 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.435512 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.436441 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.436655 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.436719 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:59.439214 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:59.439215 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:59.439215 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:59.439223 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:59.440581 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:59.440914 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:59.440915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:59.440915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:59.440970 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:59.441130 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:59.441131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:59.441131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:59.441241 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:59.441317 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:59.441400 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 22:48:59.441459 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 22:48:59.443663 17985 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:48:59.445291 17985 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:48:59.452531 18490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:59.448274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37480 (local address 127.16.121.130:32883)
0928 22:48:59.448572 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:59.448576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:59.448585 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:48:59.448621 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:59.448622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:59.448638 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:59.448711 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:48:59.448824 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.449965 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.451241 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.452069 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.452148 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:59.452202 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:59.452364 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 22:48:59.452414 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":83,"threads_started":1}
I20260928 22:48:59.452556 18488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:59.448047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46982 (local address 127.16.121.129:41783)
0928 22:48:59.448260 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:59.448265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:59.448276 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:48:59.448320 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:59.448322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:59.448339 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:59.448433 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:48:59.448754 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.449524 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.451009 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.451913 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.452004 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:59.452069 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:59.452407 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 22:48:59.452447 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":98,"threads_started":1}
I20260928 22:48:59.452592 18485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:59.447829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.130:32883 (local address 127.0.0.1:37480)
0928 22:48:59.448033 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:59.448077 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:59.448146 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:59.448708 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:59.448709 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:59.448711 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:48:59.448756 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:59.448758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.450048 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.450049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:59.451157 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:59.451159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.452153 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.452154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:59.452247 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:59.452252 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:48:59.452324 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 22:48:59.452482 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":31,"thread_start_us":89,"threads_started":1}
I20260928 22:48:59.453128 18487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:59.447829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:41783 (local address 127.0.0.1:46982)
0928 22:48:59.448033 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:59.448077 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:59.448146 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:59.448582 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:59.448586 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:59.448588 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:48:59.448664 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:59.448665 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.449607 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.449608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:59.450884 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:59.450887 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.452111 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.452113 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:59.452230 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:59.452247 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 22:48:59.452786 (+ 539us) client_negotiation.cc:240] Negotiation successful
0928 22:48:59.452841 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":3,"thread_start_us":67,"threads_started":1}
I20260928 22:48:59.453279 18486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:59.447858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.131:34453 (local address 127.0.0.1:48176)
0928 22:48:59.448382 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:59.448393 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:59.448450 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:59.448723 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:59.448724 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:59.448726 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:48:59.448774 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:59.448776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.451237 (+ 2461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.451237 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:59.451902 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:59.451904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.453047 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.453047 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:59.453119 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:59.453123 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:48:59.453178 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 22:48:59.453219 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":97,"threads_started":1}
I20260928 22:48:59.453351 18489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:59.448091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48176 (local address 127.16.121.131:34453)
0928 22:48:59.448314 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:59.448335 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:59.448446 (+ 111us) server_negotiation.cc:407] Connection header received
0928 22:48:59.448552 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:59.448554 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:59.448570 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:59.448653 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:48:59.448854 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.449958 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.451990 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.452894 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.452962 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:59.453011 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:59.453204 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 22:48:59.453242 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":68,"threads_started":1}
I20260928 22:48:59.453917 18122 tablet_service.cc:1467] Processing CreateTablet for tablet 304b5ebf4e904a7883d6e7028d710626 (DEFAULT_TABLE table=test-table [id=2b573f17dbee4580b1579fffdd33c484]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:48:59.453914 18262 tablet_service.cc:1467] Processing CreateTablet for tablet 304b5ebf4e904a7883d6e7028d710626 (DEFAULT_TABLE table=test-table [id=2b573f17dbee4580b1579fffdd33c484]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:48:59.454083 18402 tablet_service.cc:1467] Processing CreateTablet for tablet 304b5ebf4e904a7883d6e7028d710626 (DEFAULT_TABLE table=test-table [id=2b573f17dbee4580b1579fffdd33c484]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:48:59.454289 18262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 304b5ebf4e904a7883d6e7028d710626. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:59.454289 18122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 304b5ebf4e904a7883d6e7028d710626. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:59.454490 18402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 304b5ebf4e904a7883d6e7028d710626. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:59.457820 18491 tablet_bootstrap.cc:492] T 304b5ebf4e904a7883d6e7028d710626 P d5b10d16e38945e287215c52aa4c9aca: Bootstrap starting.
I20260928 22:48:59.458215 18492 tablet_bootstrap.cc:492] T 304b5ebf4e904a7883d6e7028d710626 P f168b2278558423e9b6d11ecc9314584: Bootstrap starting.
I20260928 22:48:59.459012 18492 tablet_bootstrap.cc:654] T 304b5ebf4e904a7883d6e7028d710626 P f168b2278558423e9b6d11ecc9314584: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:59.459064 18491 tablet_bootstrap.cc:654] T 304b5ebf4e904a7883d6e7028d710626 P d5b10d16e38945e287215c52aa4c9aca: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:59.459440 18492 log.cc:824] T 304b5ebf4e904a7883d6e7028d710626 P f168b2278558423e9b6d11ecc9314584: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:59.459440 18491 log.cc:824] T 304b5ebf4e904a7883d6e7028d710626 P d5b10d16e38945e287215c52aa4c9aca: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:59.459813 18493 tablet_bootstrap.cc:492] T 304b5ebf4e904a7883d6e7028d710626 P c19bcd4777c748638a4a12129d17face: Bootstrap starting.
I20260928 22:48:59.460271 18492 tablet_bootstrap.cc:492] T 304b5ebf4e904a7883d6e7028d710626 P f168b2278558423e9b6d11ecc9314584: No bootstrap required, opened a new log
I20260928 22:48:59.460355 18492 ts_tablet_manager.cc:1397] T 304b5ebf4e904a7883d6e7028d710626 P f168b2278558423e9b6d11ecc9314584: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:48:59.460480 18493 tablet_bootstrap.cc:654] T 304b5ebf4e904a7883d6e7028d710626 P c19bcd4777c748638a4a12129d17face: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:59.460530 18491 tablet_bootstrap.cc:492] T 304b5ebf4e904a7883d6e7028d710626 P d5b10d16e38945e287215c52aa4c9aca: No bootstrap required, opened a new log
I20260928 22:48:59.460589 18491 ts_tablet_manager.cc:1397] T 304b5ebf4e904a7883d6e7028d710626 P d5b10d16e38945e287215c52aa4c9aca: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 22:48:59.460857 18493 log.cc:824] T 304b5ebf4e904a7883d6e7028d710626 P c19bcd4777c748638a4a12129d17face: Log is configured to *not* fsync() on all Append() calls
I20260928 22:48:59.461978 18492 raft_consensus.cc:348] T 304b5ebf4e904a7883d6e7028d710626 P f168b2278558423e9b6d11ecc9314584 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19bcd4777c748638a4a12129d17face" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41783 } } peers { permanent_uuid: "d5b10d16e38945e287215c52aa4c9aca" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34453 } } peers { permanent_uuid: "f168b2278558423e9b6d11ecc9314584" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32883 } }
I20260928 22:48:59.462136 18492 raft_consensus.cc:374] T 304b5ebf4e904a7883d6e7028d710626 P f168b2278558423e9b6d11ecc9314584 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:59.462180 18492 raft_consensus.cc:729] T 304b5ebf4e904a7883d6e7028d710626 P f168b2278558423e9b6d11ecc9314584 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f168b2278558423e9b6d11ecc9314584, State: Initialized, Role: FOLLOWER
I20260928 22:48:59.462299 18492 consensus_queue.cc:260] T 304b5ebf4e904a7883d6e7028d710626 P f168b2278558423e9b6d11ecc9314584 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19bcd4777c748638a4a12129d17face" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41783 } } peers { permanent_uuid: "d5b10d16e38945e287215c52aa4c9aca" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34453 } } peers { permanent_uuid: "f168b2278558423e9b6d11ecc9314584" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32883 } }
I20260928 22:48:59.462368 18491 raft_consensus.cc:348] T 304b5ebf4e904a7883d6e7028d710626 P d5b10d16e38945e287215c52aa4c9aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19bcd4777c748638a4a12129d17face" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41783 } } peers { permanent_uuid: "d5b10d16e38945e287215c52aa4c9aca" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34453 } } peers { permanent_uuid: "f168b2278558423e9b6d11ecc9314584" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32883 } }
I20260928 22:48:59.462559 18491 raft_consensus.cc:374] T 304b5ebf4e904a7883d6e7028d710626 P d5b10d16e38945e287215c52aa4c9aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:59.462604 18492 ts_tablet_manager.cc:1428] T 304b5ebf4e904a7883d6e7028d710626 P f168b2278558423e9b6d11ecc9314584: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 22:48:59.462611 18491 raft_consensus.cc:729] T 304b5ebf4e904a7883d6e7028d710626 P d5b10d16e38945e287215c52aa4c9aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5b10d16e38945e287215c52aa4c9aca, State: Initialized, Role: FOLLOWER
I20260928 22:48:59.462653 18328 heartbeater.cc:503] Master 127.16.121.190:33833 was elected leader, sending a full tablet report...
I20260928 22:48:59.462749 18491 consensus_queue.cc:260] T 304b5ebf4e904a7883d6e7028d710626 P d5b10d16e38945e287215c52aa4c9aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19bcd4777c748638a4a12129d17face" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41783 } } peers { permanent_uuid: "d5b10d16e38945e287215c52aa4c9aca" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34453 } } peers { permanent_uuid: "f168b2278558423e9b6d11ecc9314584" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32883 } }
I20260928 22:48:59.463027 18491 ts_tablet_manager.cc:1428] T 304b5ebf4e904a7883d6e7028d710626 P d5b10d16e38945e287215c52aa4c9aca: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 22:48:59.463209 18468 heartbeater.cc:503] Master 127.16.121.190:33833 was elected leader, sending a full tablet report...
I20260928 22:48:59.463876 18493 tablet_bootstrap.cc:492] T 304b5ebf4e904a7883d6e7028d710626 P c19bcd4777c748638a4a12129d17face: No bootstrap required, opened a new log
I20260928 22:48:59.463974 18493 ts_tablet_manager.cc:1397] T 304b5ebf4e904a7883d6e7028d710626 P c19bcd4777c748638a4a12129d17face: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 22:48:59.465497 18493 raft_consensus.cc:348] T 304b5ebf4e904a7883d6e7028d710626 P c19bcd4777c748638a4a12129d17face [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19bcd4777c748638a4a12129d17face" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41783 } } peers { permanent_uuid: "d5b10d16e38945e287215c52aa4c9aca" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34453 } } peers { permanent_uuid: "f168b2278558423e9b6d11ecc9314584" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32883 } }
I20260928 22:48:59.465631 18493 raft_consensus.cc:374] T 304b5ebf4e904a7883d6e7028d710626 P c19bcd4777c748638a4a12129d17face [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:59.465653 18493 raft_consensus.cc:729] T 304b5ebf4e904a7883d6e7028d710626 P c19bcd4777c748638a4a12129d17face [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c19bcd4777c748638a4a12129d17face, State: Initialized, Role: FOLLOWER
I20260928 22:48:59.465767 18493 consensus_queue.cc:260] T 304b5ebf4e904a7883d6e7028d710626 P c19bcd4777c748638a4a12129d17face [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19bcd4777c748638a4a12129d17face" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41783 } } peers { permanent_uuid: "d5b10d16e38945e287215c52aa4c9aca" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34453 } } peers { permanent_uuid: "f168b2278558423e9b6d11ecc9314584" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32883 } }
I20260928 22:48:59.466004 18493 ts_tablet_manager.cc:1428] T 304b5ebf4e904a7883d6e7028d710626 P c19bcd4777c748638a4a12129d17face: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:48:59.466056 18188 heartbeater.cc:503] Master 127.16.121.190:33833 was elected leader, sending a full tablet report...
W20260928 22:48:59.482121 18329 tablet.cc:2369] T 304b5ebf4e904a7883d6e7028d710626 P f168b2278558423e9b6d11ecc9314584: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:48:59.564425 18189 tablet.cc:2369] T 304b5ebf4e904a7883d6e7028d710626 P c19bcd4777c748638a4a12129d17face: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:48:59.634505 18498 raft_consensus.cc:482] T 304b5ebf4e904a7883d6e7028d710626 P d5b10d16e38945e287215c52aa4c9aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:48:59.634646 18498 raft_consensus.cc:504] T 304b5ebf4e904a7883d6e7028d710626 P d5b10d16e38945e287215c52aa4c9aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19bcd4777c748638a4a12129d17face" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41783 } } peers { permanent_uuid: "d5b10d16e38945e287215c52aa4c9aca" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34453 } } peers { permanent_uuid: "f168b2278558423e9b6d11ecc9314584" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32883 } }
I20260928 22:48:59.635162 18498 leader_election.cc:288] T 304b5ebf4e904a7883d6e7028d710626 P d5b10d16e38945e287215c52aa4c9aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c19bcd4777c748638a4a12129d17face (127.16.121.129:41783), f168b2278558423e9b6d11ecc9314584 (127.16.121.130:32883)
I20260928 22:48:59.638273 18490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:59.635586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:36737 (local address 127.16.121.130:32883)
0928 22:48:59.635654 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:59.635656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:59.635665 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:48:59.635692 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:59.635693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:59.635702 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:59.635769 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:48:59.636053 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.636609 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.637519 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.637874 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.637963 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:59.637974 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:59.638143 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 22:48:59.638186 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 22:48:59.638267 18500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:59.635405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.130:32883 (local address 127.16.121.131:36737)
0928 22:48:59.635593 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:59.635603 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:59.635647 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:59.635844 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:59.635845 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:59.635864 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:48:59.635975 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:59.635977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.636688 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.636689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:59.637407 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:59.637409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.637959 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.637960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:59.638069 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:59.638073 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:48:59.638116 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 22:48:59.638177 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":117,"threads_started":1}
I20260928 22:48:59.638453 18488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:59.635451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:56313 (local address 127.16.121.129:41783)
0928 22:48:59.635502 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:59.635504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:59.635518 (+ 14us) server_negotiation.cc:407] Connection header received
0928 22:48:59.635557 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:59.635558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:59.635569 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:59.635636 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:48:59.635840 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.636441 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.637473 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.637876 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.637966 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:59.637976 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:59.638311 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 22:48:59.638370 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 22:48:59.638476 18361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:59.635382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:41783 (local address 127.16.121.131:56313)
0928 22:48:59.635435 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:59.635447 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:59.635493 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:59.635667 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:59.635669 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:59.635671 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:48:59.635748 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:59.635750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.636526 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.636527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:59.637375 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:59.637377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.638040 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.638041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:59.638162 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:59.638189 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 22:48:59.638280 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 22:48:59.638342 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":21}
I20260928 22:48:59.638847 18282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "304b5ebf4e904a7883d6e7028d710626" candidate_uuid: "d5b10d16e38945e287215c52aa4c9aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f168b2278558423e9b6d11ecc9314584" is_pre_election: true
I20260928 22:48:59.639019 18282 raft_consensus.cc:2454] T 304b5ebf4e904a7883d6e7028d710626 P f168b2278558423e9b6d11ecc9314584 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5b10d16e38945e287215c52aa4c9aca in term 0.
I20260928 22:48:59.639021 18142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "304b5ebf4e904a7883d6e7028d710626" candidate_uuid: "d5b10d16e38945e287215c52aa4c9aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c19bcd4777c748638a4a12129d17face" is_pre_election: true
I20260928 22:48:59.639231 18142 raft_consensus.cc:2454] T 304b5ebf4e904a7883d6e7028d710626 P c19bcd4777c748638a4a12129d17face [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5b10d16e38945e287215c52aa4c9aca in term 0.
I20260928 22:48:59.639307 18353 leader_election.cc:302] T 304b5ebf4e904a7883d6e7028d710626 P d5b10d16e38945e287215c52aa4c9aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5b10d16e38945e287215c52aa4c9aca, f168b2278558423e9b6d11ecc9314584; no voters:
I20260928 22:48:59.639463 18498 raft_consensus.cc:2781] T 304b5ebf4e904a7883d6e7028d710626 P d5b10d16e38945e287215c52aa4c9aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:48:59.639514 18498 raft_consensus.cc:482] T 304b5ebf4e904a7883d6e7028d710626 P d5b10d16e38945e287215c52aa4c9aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:48:59.639532 18498 raft_consensus.cc:3037] T 304b5ebf4e904a7883d6e7028d710626 P d5b10d16e38945e287215c52aa4c9aca [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:59.640273 18498 raft_consensus.cc:504] T 304b5ebf4e904a7883d6e7028d710626 P d5b10d16e38945e287215c52aa4c9aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19bcd4777c748638a4a12129d17face" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41783 } } peers { permanent_uuid: "d5b10d16e38945e287215c52aa4c9aca" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34453 } } peers { permanent_uuid: "f168b2278558423e9b6d11ecc9314584" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32883 } }
I20260928 22:48:59.640424 18498 leader_election.cc:288] T 304b5ebf4e904a7883d6e7028d710626 P d5b10d16e38945e287215c52aa4c9aca [CANDIDATE]: Term 1 election: Requested vote from peers c19bcd4777c748638a4a12129d17face (127.16.121.129:41783), f168b2278558423e9b6d11ecc9314584 (127.16.121.130:32883)
I20260928 22:48:59.640575 18142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "304b5ebf4e904a7883d6e7028d710626" candidate_uuid: "d5b10d16e38945e287215c52aa4c9aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c19bcd4777c748638a4a12129d17face"
I20260928 22:48:59.640623 18282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "304b5ebf4e904a7883d6e7028d710626" candidate_uuid: "d5b10d16e38945e287215c52aa4c9aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f168b2278558423e9b6d11ecc9314584"
I20260928 22:48:59.640655 18142 raft_consensus.cc:3037] T 304b5ebf4e904a7883d6e7028d710626 P c19bcd4777c748638a4a12129d17face [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:59.640691 18282 raft_consensus.cc:3037] T 304b5ebf4e904a7883d6e7028d710626 P f168b2278558423e9b6d11ecc9314584 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:59.641562 18282 raft_consensus.cc:2454] T 304b5ebf4e904a7883d6e7028d710626 P f168b2278558423e9b6d11ecc9314584 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b10d16e38945e287215c52aa4c9aca in term 1.
I20260928 22:48:59.641562 18142 raft_consensus.cc:2454] T 304b5ebf4e904a7883d6e7028d710626 P c19bcd4777c748638a4a12129d17face [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b10d16e38945e287215c52aa4c9aca in term 1.
I20260928 22:48:59.642104 18356 leader_election.cc:302] T 304b5ebf4e904a7883d6e7028d710626 P d5b10d16e38945e287215c52aa4c9aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c19bcd4777c748638a4a12129d17face, d5b10d16e38945e287215c52aa4c9aca; no voters:
I20260928 22:48:59.642222 18498 raft_consensus.cc:2781] T 304b5ebf4e904a7883d6e7028d710626 P d5b10d16e38945e287215c52aa4c9aca [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:48:59.642403 18498 raft_consensus.cc:686] T 304b5ebf4e904a7883d6e7028d710626 P d5b10d16e38945e287215c52aa4c9aca [term 1 LEADER]: Becoming Leader. State: Replica: d5b10d16e38945e287215c52aa4c9aca, State: Running, Role: LEADER
I20260928 22:48:59.642516 18498 consensus_queue.cc:237] T 304b5ebf4e904a7883d6e7028d710626 P d5b10d16e38945e287215c52aa4c9aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19bcd4777c748638a4a12129d17face" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41783 } } peers { permanent_uuid: "d5b10d16e38945e287215c52aa4c9aca" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34453 } } peers { permanent_uuid: "f168b2278558423e9b6d11ecc9314584" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32883 } }
I20260928 22:48:59.643376 17985 catalog_manager.cc:5495] T 304b5ebf4e904a7883d6e7028d710626 P d5b10d16e38945e287215c52aa4c9aca reported cstate change: term changed from 0 to 1, leader changed from <none> to d5b10d16e38945e287215c52aa4c9aca (127.16.121.131). New cstate: current_term: 1 leader_uuid: "d5b10d16e38945e287215c52aa4c9aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19bcd4777c748638a4a12129d17face" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5b10d16e38945e287215c52aa4c9aca" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f168b2278558423e9b6d11ecc9314584" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32883 } health_report { overall_health: UNKNOWN } } }
W20260928 22:48:59.653374 18469 tablet.cc:2369] T 304b5ebf4e904a7883d6e7028d710626 P d5b10d16e38945e287215c52aa4c9aca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:48:59.681221 18489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:59.678301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48188 (local address 127.16.121.131:34453)
0928 22:48:59.678363 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:59.678365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:59.678374 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:48:59.678419 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:59.678420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:59.678437 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:59.678510 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:48:59.678694 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.679298 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.680230 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.680467 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.680552 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:59.680788 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:48:59.680913 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:48:59.681009 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:59.681073 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 22:48:59.681098 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 22:48:59.683482 18282 raft_consensus.cc:1261] T 304b5ebf4e904a7883d6e7028d710626 P f168b2278558423e9b6d11ecc9314584 [term 1 FOLLOWER]: Refusing update from remote peer d5b10d16e38945e287215c52aa4c9aca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:48:59.683456 18142 raft_consensus.cc:1261] T 304b5ebf4e904a7883d6e7028d710626 P c19bcd4777c748638a4a12129d17face [term 1 FOLLOWER]: Refusing update from remote peer d5b10d16e38945e287215c52aa4c9aca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:48:59.683867 18498 consensus_queue.cc:1035] T 304b5ebf4e904a7883d6e7028d710626 P d5b10d16e38945e287215c52aa4c9aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "f168b2278558423e9b6d11ecc9314584" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:48:59.684046 18501 consensus_queue.cc:1035] T 304b5ebf4e904a7883d6e7028d710626 P d5b10d16e38945e287215c52aa4c9aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "c19bcd4777c748638a4a12129d17face" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:48:59.687346 18505 mvcc.cc:205] Tried to move back new op lower bound from 7334443989740052480 to 7334443989576441856. Current Snapshot: MvccSnapshot[applied={T|T < 7334443989740052480 or (T in {7334443989740052480})}]
I20260928 22:48:59.687541 18504 mvcc.cc:205] Tried to move back new op lower bound from 7334443989740052480 to 7334443989576441856. Current Snapshot: MvccSnapshot[applied={T|T < 7334443989740052480 or (T in {7334443989740052480})}]
I20260928 22:48:59.690042 17985 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:48:59.690127 17985 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:48:59.690930 17985 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 545769a425e048d9a2cb5f05d46f2f68: Sending DeleteTablet for 3 replicas of tablet 304b5ebf4e904a7883d6e7028d710626
I20260928 22:48:59.691255 18402 tablet_service.cc:1514] Processing DeleteTablet for tablet 304b5ebf4e904a7883d6e7028d710626 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:48:59 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:48176
I20260928 22:48:59.691272 18122 tablet_service.cc:1514] Processing DeleteTablet for tablet 304b5ebf4e904a7883d6e7028d710626 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:48:59 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:46982
I20260928 22:48:59.691490 18515 tablet_replica.cc:330] T 304b5ebf4e904a7883d6e7028d710626 P d5b10d16e38945e287215c52aa4c9aca: stopping tablet replica
I20260928 22:48:59.691651 18515 raft_consensus.cc:2229] T 304b5ebf4e904a7883d6e7028d710626 P d5b10d16e38945e287215c52aa4c9aca [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:48:59.691624 18262 tablet_service.cc:1514] Processing DeleteTablet for tablet 304b5ebf4e904a7883d6e7028d710626 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:48:59 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:37480
I20260928 22:48:59.691841 18515 raft_consensus.cc:2258] T 304b5ebf4e904a7883d6e7028d710626 P d5b10d16e38945e287215c52aa4c9aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:48:59.691802 17985 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:48:59.691953 17985 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:48:59.692245 18514 tablet_replica.cc:330] T 304b5ebf4e904a7883d6e7028d710626 P c19bcd4777c748638a4a12129d17face: stopping tablet replica
I20260928 22:48:59.692333 18515 ts_tablet_manager.cc:1905] T 304b5ebf4e904a7883d6e7028d710626 P d5b10d16e38945e287215c52aa4c9aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:48:59.693418 18402 tablet_service.cc:1467] Processing CreateTablet for tablet 00ab988c95f644ba8defff5ed10dd6c9 (DEFAULT_TABLE table=test-table [id=66798ae363df4c2c9322901e2209a01c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:48:59.693527 18402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00ab988c95f644ba8defff5ed10dd6c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:59.693522 18262 tablet_service.cc:1467] Processing CreateTablet for tablet 00ab988c95f644ba8defff5ed10dd6c9 (DEFAULT_TABLE table=test-table [id=66798ae363df4c2c9322901e2209a01c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:48:59.693634 18262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00ab988c95f644ba8defff5ed10dd6c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:59.694190 18515 ts_tablet_manager.cc:1918] T 304b5ebf4e904a7883d6e7028d710626 P d5b10d16e38945e287215c52aa4c9aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:48:59.694257 18515 log.cc:1196] T 304b5ebf4e904a7883d6e7028d710626 P d5b10d16e38945e287215c52aa4c9aca: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-2/wal/wals/304b5ebf4e904a7883d6e7028d710626
I20260928 22:48:59.694536 18515 ts_tablet_manager.cc:1939] T 304b5ebf4e904a7883d6e7028d710626 P d5b10d16e38945e287215c52aa4c9aca: Deleting consensus metadata
I20260928 22:48:59.694892 18491 tablet_bootstrap.cc:492] T 00ab988c95f644ba8defff5ed10dd6c9 P d5b10d16e38945e287215c52aa4c9aca: Bootstrap starting.
I20260928 22:48:59.695322 18491 tablet_bootstrap.cc:654] T 00ab988c95f644ba8defff5ed10dd6c9 P d5b10d16e38945e287215c52aa4c9aca: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:59.695748 18492 tablet_bootstrap.cc:492] T 00ab988c95f644ba8defff5ed10dd6c9 P f168b2278558423e9b6d11ecc9314584: Bootstrap starting.
I20260928 22:48:59.696002 18491 tablet_bootstrap.cc:492] T 00ab988c95f644ba8defff5ed10dd6c9 P d5b10d16e38945e287215c52aa4c9aca: No bootstrap required, opened a new log
I20260928 22:48:59.696039 18491 ts_tablet_manager.cc:1397] T 00ab988c95f644ba8defff5ed10dd6c9 P d5b10d16e38945e287215c52aa4c9aca: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 22:48:59.696165 18492 tablet_bootstrap.cc:654] T 00ab988c95f644ba8defff5ed10dd6c9 P f168b2278558423e9b6d11ecc9314584: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:59.696189 18491 raft_consensus.cc:348] T 00ab988c95f644ba8defff5ed10dd6c9 P d5b10d16e38945e287215c52aa4c9aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19bcd4777c748638a4a12129d17face" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41783 } } peers { permanent_uuid: "f168b2278558423e9b6d11ecc9314584" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32883 } } peers { permanent_uuid: "d5b10d16e38945e287215c52aa4c9aca" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34453 } }
I20260928 22:48:59.696261 18491 raft_consensus.cc:374] T 00ab988c95f644ba8defff5ed10dd6c9 P d5b10d16e38945e287215c52aa4c9aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:59.696286 18491 raft_consensus.cc:729] T 00ab988c95f644ba8defff5ed10dd6c9 P d5b10d16e38945e287215c52aa4c9aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5b10d16e38945e287215c52aa4c9aca, State: Initialized, Role: FOLLOWER
I20260928 22:48:59.696333 18491 consensus_queue.cc:260] T 00ab988c95f644ba8defff5ed10dd6c9 P d5b10d16e38945e287215c52aa4c9aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19bcd4777c748638a4a12129d17face" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41783 } } peers { permanent_uuid: "f168b2278558423e9b6d11ecc9314584" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32883 } } peers { permanent_uuid: "d5b10d16e38945e287215c52aa4c9aca" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34453 } }
I20260928 22:48:59.696441 18491 ts_tablet_manager.cc:1428] T 00ab988c95f644ba8defff5ed10dd6c9 P d5b10d16e38945e287215c52aa4c9aca: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:59.696780 17970 catalog_manager.cc:4841] TS d5b10d16e38945e287215c52aa4c9aca (127.16.121.131:34453): tablet 304b5ebf4e904a7883d6e7028d710626 (table test-table [id=2b573f17dbee4580b1579fffdd33c484]) successfully deleted
I20260928 22:48:59.697021 18516 tablet_replica.cc:330] T 304b5ebf4e904a7883d6e7028d710626 P f168b2278558423e9b6d11ecc9314584: stopping tablet replica
I20260928 22:48:59.697148 18516 raft_consensus.cc:2229] T 304b5ebf4e904a7883d6e7028d710626 P f168b2278558423e9b6d11ecc9314584 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:48:59.697291 18516 raft_consensus.cc:2258] T 304b5ebf4e904a7883d6e7028d710626 P f168b2278558423e9b6d11ecc9314584 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:48:59.693257 18122 tablet_service.cc:1467] Processing CreateTablet for tablet 00ab988c95f644ba8defff5ed10dd6c9 (DEFAULT_TABLE table=test-table [id=66798ae363df4c2c9322901e2209a01c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:48:59.697500 18122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00ab988c95f644ba8defff5ed10dd6c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:59.697484 18514 raft_consensus.cc:2229] T 304b5ebf4e904a7883d6e7028d710626 P c19bcd4777c748638a4a12129d17face [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:48:59.697664 18514 raft_consensus.cc:2258] T 304b5ebf4e904a7883d6e7028d710626 P c19bcd4777c748638a4a12129d17face [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:48:59.697827 18492 tablet_bootstrap.cc:492] T 00ab988c95f644ba8defff5ed10dd6c9 P f168b2278558423e9b6d11ecc9314584: No bootstrap required, opened a new log
I20260928 22:48:59.697867 18492 ts_tablet_manager.cc:1397] T 00ab988c95f644ba8defff5ed10dd6c9 P f168b2278558423e9b6d11ecc9314584: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:48:59.698019 18492 raft_consensus.cc:348] T 00ab988c95f644ba8defff5ed10dd6c9 P f168b2278558423e9b6d11ecc9314584 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19bcd4777c748638a4a12129d17face" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41783 } } peers { permanent_uuid: "f168b2278558423e9b6d11ecc9314584" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32883 } } peers { permanent_uuid: "d5b10d16e38945e287215c52aa4c9aca" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34453 } }
I20260928 22:48:59.698077 18492 raft_consensus.cc:374] T 00ab988c95f644ba8defff5ed10dd6c9 P f168b2278558423e9b6d11ecc9314584 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:59.698096 18492 raft_consensus.cc:729] T 00ab988c95f644ba8defff5ed10dd6c9 P f168b2278558423e9b6d11ecc9314584 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f168b2278558423e9b6d11ecc9314584, State: Initialized, Role: FOLLOWER
I20260928 22:48:59.698145 18492 consensus_queue.cc:260] T 00ab988c95f644ba8defff5ed10dd6c9 P f168b2278558423e9b6d11ecc9314584 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19bcd4777c748638a4a12129d17face" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41783 } } peers { permanent_uuid: "f168b2278558423e9b6d11ecc9314584" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32883 } } peers { permanent_uuid: "d5b10d16e38945e287215c52aa4c9aca" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34453 } }
I20260928 22:48:59.698239 18492 ts_tablet_manager.cc:1428] T 00ab988c95f644ba8defff5ed10dd6c9 P f168b2278558423e9b6d11ecc9314584: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 22:48:59.698961 18493 tablet_bootstrap.cc:492] T 00ab988c95f644ba8defff5ed10dd6c9 P c19bcd4777c748638a4a12129d17face: Bootstrap starting.
I20260928 22:48:59.699369 18493 tablet_bootstrap.cc:654] T 00ab988c95f644ba8defff5ed10dd6c9 P c19bcd4777c748638a4a12129d17face: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:59.699564 18514 ts_tablet_manager.cc:1905] T 304b5ebf4e904a7883d6e7028d710626 P c19bcd4777c748638a4a12129d17face: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:48:59.699753 18516 ts_tablet_manager.cc:1905] T 304b5ebf4e904a7883d6e7028d710626 P f168b2278558423e9b6d11ecc9314584: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:48:59.700222 18493 tablet_bootstrap.cc:492] T 00ab988c95f644ba8defff5ed10dd6c9 P c19bcd4777c748638a4a12129d17face: No bootstrap required, opened a new log
I20260928 22:48:59.700268 18493 ts_tablet_manager.cc:1397] T 00ab988c95f644ba8defff5ed10dd6c9 P c19bcd4777c748638a4a12129d17face: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:48:59.700414 18493 raft_consensus.cc:348] T 00ab988c95f644ba8defff5ed10dd6c9 P c19bcd4777c748638a4a12129d17face [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19bcd4777c748638a4a12129d17face" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41783 } } peers { permanent_uuid: "f168b2278558423e9b6d11ecc9314584" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32883 } } peers { permanent_uuid: "d5b10d16e38945e287215c52aa4c9aca" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34453 } }
I20260928 22:48:59.700488 18493 raft_consensus.cc:374] T 00ab988c95f644ba8defff5ed10dd6c9 P c19bcd4777c748638a4a12129d17face [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:59.700510 18493 raft_consensus.cc:729] T 00ab988c95f644ba8defff5ed10dd6c9 P c19bcd4777c748638a4a12129d17face [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c19bcd4777c748638a4a12129d17face, State: Initialized, Role: FOLLOWER
I20260928 22:48:59.700551 18493 consensus_queue.cc:260] T 00ab988c95f644ba8defff5ed10dd6c9 P c19bcd4777c748638a4a12129d17face [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19bcd4777c748638a4a12129d17face" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41783 } } peers { permanent_uuid: "f168b2278558423e9b6d11ecc9314584" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32883 } } peers { permanent_uuid: "d5b10d16e38945e287215c52aa4c9aca" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34453 } }
I20260928 22:48:59.700654 18493 ts_tablet_manager.cc:1428] T 00ab988c95f644ba8defff5ed10dd6c9 P c19bcd4777c748638a4a12129d17face: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:59.701447 18516 ts_tablet_manager.cc:1918] T 304b5ebf4e904a7883d6e7028d710626 P f168b2278558423e9b6d11ecc9314584: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:48:59.701462 18514 ts_tablet_manager.cc:1918] T 304b5ebf4e904a7883d6e7028d710626 P c19bcd4777c748638a4a12129d17face: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:48:59.701532 18514 log.cc:1196] T 304b5ebf4e904a7883d6e7028d710626 P c19bcd4777c748638a4a12129d17face: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-0/wal/wals/304b5ebf4e904a7883d6e7028d710626
I20260928 22:48:59.701535 18516 log.cc:1196] T 304b5ebf4e904a7883d6e7028d710626 P f168b2278558423e9b6d11ecc9314584: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-1/wal/wals/304b5ebf4e904a7883d6e7028d710626
I20260928 22:48:59.701776 18516 ts_tablet_manager.cc:1939] T 304b5ebf4e904a7883d6e7028d710626 P f168b2278558423e9b6d11ecc9314584: Deleting consensus metadata
I20260928 22:48:59.701849 18514 ts_tablet_manager.cc:1939] T 304b5ebf4e904a7883d6e7028d710626 P c19bcd4777c748638a4a12129d17face: Deleting consensus metadata
I20260928 22:48:59.702533 17969 catalog_manager.cc:4841] TS f168b2278558423e9b6d11ecc9314584 (127.16.121.130:32883): tablet 304b5ebf4e904a7883d6e7028d710626 (table test-table [id=2b573f17dbee4580b1579fffdd33c484]) successfully deleted
I20260928 22:48:59.702648 17972 catalog_manager.cc:4841] TS c19bcd4777c748638a4a12129d17face (127.16.121.129:41783): tablet 304b5ebf4e904a7883d6e7028d710626 (table test-table [id=2b573f17dbee4580b1579fffdd33c484]) successfully deleted
I20260928 22:48:59.910467 18509 raft_consensus.cc:482] T 00ab988c95f644ba8defff5ed10dd6c9 P d5b10d16e38945e287215c52aa4c9aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:48:59.910772 18509 raft_consensus.cc:504] T 00ab988c95f644ba8defff5ed10dd6c9 P d5b10d16e38945e287215c52aa4c9aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19bcd4777c748638a4a12129d17face" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41783 } } peers { permanent_uuid: "f168b2278558423e9b6d11ecc9314584" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32883 } } peers { permanent_uuid: "d5b10d16e38945e287215c52aa4c9aca" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34453 } }
I20260928 22:48:59.911016 18509 leader_election.cc:288] T 00ab988c95f644ba8defff5ed10dd6c9 P d5b10d16e38945e287215c52aa4c9aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c19bcd4777c748638a4a12129d17face (127.16.121.129:41783), f168b2278558423e9b6d11ecc9314584 (127.16.121.130:32883)
I20260928 22:48:59.911293 18142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ab988c95f644ba8defff5ed10dd6c9" candidate_uuid: "d5b10d16e38945e287215c52aa4c9aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c19bcd4777c748638a4a12129d17face" is_pre_election: true
I20260928 22:48:59.911401 18142 raft_consensus.cc:2454] T 00ab988c95f644ba8defff5ed10dd6c9 P c19bcd4777c748638a4a12129d17face [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5b10d16e38945e287215c52aa4c9aca in term 0.
I20260928 22:48:59.911348 18282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ab988c95f644ba8defff5ed10dd6c9" candidate_uuid: "d5b10d16e38945e287215c52aa4c9aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f168b2278558423e9b6d11ecc9314584" is_pre_election: true
I20260928 22:48:59.911470 18282 raft_consensus.cc:2454] T 00ab988c95f644ba8defff5ed10dd6c9 P f168b2278558423e9b6d11ecc9314584 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5b10d16e38945e287215c52aa4c9aca in term 0.
I20260928 22:48:59.911625 18356 leader_election.cc:302] T 00ab988c95f644ba8defff5ed10dd6c9 P d5b10d16e38945e287215c52aa4c9aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c19bcd4777c748638a4a12129d17face, d5b10d16e38945e287215c52aa4c9aca; no voters:
I20260928 22:48:59.911769 18509 raft_consensus.cc:2781] T 00ab988c95f644ba8defff5ed10dd6c9 P d5b10d16e38945e287215c52aa4c9aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:48:59.911815 18509 raft_consensus.cc:482] T 00ab988c95f644ba8defff5ed10dd6c9 P d5b10d16e38945e287215c52aa4c9aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:48:59.911852 18509 raft_consensus.cc:3037] T 00ab988c95f644ba8defff5ed10dd6c9 P d5b10d16e38945e287215c52aa4c9aca [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:59.912873 18509 raft_consensus.cc:504] T 00ab988c95f644ba8defff5ed10dd6c9 P d5b10d16e38945e287215c52aa4c9aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19bcd4777c748638a4a12129d17face" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41783 } } peers { permanent_uuid: "f168b2278558423e9b6d11ecc9314584" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32883 } } peers { permanent_uuid: "d5b10d16e38945e287215c52aa4c9aca" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34453 } }
I20260928 22:48:59.913067 18509 leader_election.cc:288] T 00ab988c95f644ba8defff5ed10dd6c9 P d5b10d16e38945e287215c52aa4c9aca [CANDIDATE]: Term 1 election: Requested vote from peers c19bcd4777c748638a4a12129d17face (127.16.121.129:41783), f168b2278558423e9b6d11ecc9314584 (127.16.121.130:32883)
I20260928 22:48:59.913264 18142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ab988c95f644ba8defff5ed10dd6c9" candidate_uuid: "d5b10d16e38945e287215c52aa4c9aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c19bcd4777c748638a4a12129d17face"
I20260928 22:48:59.913372 18142 raft_consensus.cc:3037] T 00ab988c95f644ba8defff5ed10dd6c9 P c19bcd4777c748638a4a12129d17face [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:59.913496 18282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ab988c95f644ba8defff5ed10dd6c9" candidate_uuid: "d5b10d16e38945e287215c52aa4c9aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f168b2278558423e9b6d11ecc9314584"
I20260928 22:48:59.913590 18282 raft_consensus.cc:3037] T 00ab988c95f644ba8defff5ed10dd6c9 P f168b2278558423e9b6d11ecc9314584 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:48:59.914103 18142 raft_consensus.cc:2454] T 00ab988c95f644ba8defff5ed10dd6c9 P c19bcd4777c748638a4a12129d17face [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b10d16e38945e287215c52aa4c9aca in term 1.
I20260928 22:48:59.914264 18282 raft_consensus.cc:2454] T 00ab988c95f644ba8defff5ed10dd6c9 P f168b2278558423e9b6d11ecc9314584 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b10d16e38945e287215c52aa4c9aca in term 1.
I20260928 22:48:59.914299 18356 leader_election.cc:302] T 00ab988c95f644ba8defff5ed10dd6c9 P d5b10d16e38945e287215c52aa4c9aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c19bcd4777c748638a4a12129d17face, d5b10d16e38945e287215c52aa4c9aca; no voters:
I20260928 22:48:59.914460 18509 raft_consensus.cc:2781] T 00ab988c95f644ba8defff5ed10dd6c9 P d5b10d16e38945e287215c52aa4c9aca [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:48:59.914537 18509 raft_consensus.cc:686] T 00ab988c95f644ba8defff5ed10dd6c9 P d5b10d16e38945e287215c52aa4c9aca [term 1 LEADER]: Becoming Leader. State: Replica: d5b10d16e38945e287215c52aa4c9aca, State: Running, Role: LEADER
I20260928 22:48:59.914611 18509 consensus_queue.cc:237] T 00ab988c95f644ba8defff5ed10dd6c9 P d5b10d16e38945e287215c52aa4c9aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19bcd4777c748638a4a12129d17face" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41783 } } peers { permanent_uuid: "f168b2278558423e9b6d11ecc9314584" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32883 } } peers { permanent_uuid: "d5b10d16e38945e287215c52aa4c9aca" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34453 } }
I20260928 22:48:59.915366 17985 catalog_manager.cc:5495] T 00ab988c95f644ba8defff5ed10dd6c9 P d5b10d16e38945e287215c52aa4c9aca reported cstate change: term changed from 0 to 1, leader changed from <none> to d5b10d16e38945e287215c52aa4c9aca (127.16.121.131). New cstate: current_term: 1 leader_uuid: "d5b10d16e38945e287215c52aa4c9aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19bcd4777c748638a4a12129d17face" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f168b2278558423e9b6d11ecc9314584" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5b10d16e38945e287215c52aa4c9aca" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34453 } health_report { overall_health: HEALTHY } } }
Sep 28 22:48:59 dist-test-slave-56vw krb5kdc[17940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635738, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:48:59.951006 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:59.943893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40762 (local address 127.16.121.190:33833)
0928 22:48:59.944115 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 22:48:59.944118 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:48:59.944128 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:48:59.944197 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:48:59.944198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:48:59.944209 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:48:59.944290 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:48:59.944776 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.945294 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.946002 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.946210 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.946293 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:59.948433 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:48:59.948434 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:48:59.948435 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:48:59.948442 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:48:59.949751 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:59.950161 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:59.950162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:59.950163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:59.950216 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:48:59.950429 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:48:59.950430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:48:59.950430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:48:59.950539 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:48:59.950611 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 22:48:59.950765 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 22:48:59.950882 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":146,"thread_start_us":107,"threads_started":1}
I20260928 22:48:59.951090 18526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:48:59.943823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:33833 (local address 127.0.0.1:40762)
0928 22:48:59.943986 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 22:48:59.944019 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 22:48:59.944125 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:48:59.944293 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:48:59.944294 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:48:59.944631 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:48:59.944680 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:59.944682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.945376 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.945377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:59.945919 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:48:59.945921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:48:59.946296 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:48:59.946296 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:48:59.946346 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:48:59.946894 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:48:59.946899 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:48:59.948343 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:48:59.949844 (+ 1501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:59.949851 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:59.949854 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:59.950076 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:59.950296 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:48:59.950297 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:48:59.950297 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:48:59.950355 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:48:59.950601 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:48:59.950608 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:48:59.950654 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 22:48:59.950730 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 22:48:59.950944 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":98,"thread_start_us":94,"threads_started":1}
I20260928 22:48:59.953404 17985 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:40762:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 22:48:59.953642 17985 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:48:59.955230 18402 tablet_service.cc:1467] Processing CreateTablet for tablet 8942dea468384e7d83ff1a7890118deb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ea3d797c597346f8a6ea2339d8adb1b3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 22:48:59.955219 18262 tablet_service.cc:1467] Processing CreateTablet for tablet 8942dea468384e7d83ff1a7890118deb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ea3d797c597346f8a6ea2339d8adb1b3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 22:48:59.955245 18122 tablet_service.cc:1467] Processing CreateTablet for tablet 8942dea468384e7d83ff1a7890118deb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ea3d797c597346f8a6ea2339d8adb1b3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 22:48:59.955392 18402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8942dea468384e7d83ff1a7890118deb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:59.955402 18262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8942dea468384e7d83ff1a7890118deb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:59.955437 18122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8942dea468384e7d83ff1a7890118deb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:48:59.956833 18493 tablet_bootstrap.cc:492] T 8942dea468384e7d83ff1a7890118deb P c19bcd4777c748638a4a12129d17face: Bootstrap starting.
I20260928 22:48:59.956931 18492 tablet_bootstrap.cc:492] T 8942dea468384e7d83ff1a7890118deb P f168b2278558423e9b6d11ecc9314584: Bootstrap starting.
I20260928 22:48:59.957231 18493 tablet_bootstrap.cc:654] T 8942dea468384e7d83ff1a7890118deb P c19bcd4777c748638a4a12129d17face: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:59.957268 18492 tablet_bootstrap.cc:654] T 8942dea468384e7d83ff1a7890118deb P f168b2278558423e9b6d11ecc9314584: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:59.957595 18491 tablet_bootstrap.cc:492] T 8942dea468384e7d83ff1a7890118deb P d5b10d16e38945e287215c52aa4c9aca: Bootstrap starting.
I20260928 22:48:59.957806 18493 tablet_bootstrap.cc:492] T 8942dea468384e7d83ff1a7890118deb P c19bcd4777c748638a4a12129d17face: No bootstrap required, opened a new log
I20260928 22:48:59.957840 18493 ts_tablet_manager.cc:1397] T 8942dea468384e7d83ff1a7890118deb P c19bcd4777c748638a4a12129d17face: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:48:59.957927 18491 tablet_bootstrap.cc:654] T 8942dea468384e7d83ff1a7890118deb P d5b10d16e38945e287215c52aa4c9aca: Neither blocks nor log segments found. Creating new log.
I20260928 22:48:59.957952 18493 raft_consensus.cc:348] T 8942dea468384e7d83ff1a7890118deb P c19bcd4777c748638a4a12129d17face [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19bcd4777c748638a4a12129d17face" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41783 } } peers { permanent_uuid: "f168b2278558423e9b6d11ecc9314584" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32883 } } peers { permanent_uuid: "d5b10d16e38945e287215c52aa4c9aca" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34453 } }
I20260928 22:48:59.958003 18493 raft_consensus.cc:374] T 8942dea468384e7d83ff1a7890118deb P c19bcd4777c748638a4a12129d17face [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:59.958017 18493 raft_consensus.cc:729] T 8942dea468384e7d83ff1a7890118deb P c19bcd4777c748638a4a12129d17face [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c19bcd4777c748638a4a12129d17face, State: Initialized, Role: FOLLOWER
I20260928 22:48:59.958073 18493 consensus_queue.cc:260] T 8942dea468384e7d83ff1a7890118deb P c19bcd4777c748638a4a12129d17face [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19bcd4777c748638a4a12129d17face" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41783 } } peers { permanent_uuid: "f168b2278558423e9b6d11ecc9314584" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32883 } } peers { permanent_uuid: "d5b10d16e38945e287215c52aa4c9aca" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34453 } }
I20260928 22:48:59.958204 18493 ts_tablet_manager.cc:1428] T 8942dea468384e7d83ff1a7890118deb P c19bcd4777c748638a4a12129d17face: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:59.958247 18499 tablet_replica.cc:439] T 8942dea468384e7d83ff1a7890118deb P c19bcd4777c748638a4a12129d17face: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19bcd4777c748638a4a12129d17face" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41783 } } peers { permanent_uuid: "f168b2278558423e9b6d11ecc9314584" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32883 } } peers { permanent_uuid: "d5b10d16e38945e287215c52aa4c9aca" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34453 } } }
I20260928 22:48:59.958341 18499 tablet_replica.cc:442] T 8942dea468384e7d83ff1a7890118deb P c19bcd4777c748638a4a12129d17face: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:48:59.958451 18491 tablet_bootstrap.cc:492] T 8942dea468384e7d83ff1a7890118deb P d5b10d16e38945e287215c52aa4c9aca: No bootstrap required, opened a new log
I20260928 22:48:59.958506 18491 ts_tablet_manager.cc:1397] T 8942dea468384e7d83ff1a7890118deb P d5b10d16e38945e287215c52aa4c9aca: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 22:48:59.958631 18491 raft_consensus.cc:348] T 8942dea468384e7d83ff1a7890118deb P d5b10d16e38945e287215c52aa4c9aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19bcd4777c748638a4a12129d17face" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41783 } } peers { permanent_uuid: "f168b2278558423e9b6d11ecc9314584" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32883 } } peers { permanent_uuid: "d5b10d16e38945e287215c52aa4c9aca" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34453 } }
I20260928 22:48:59.958690 18491 raft_consensus.cc:374] T 8942dea468384e7d83ff1a7890118deb P d5b10d16e38945e287215c52aa4c9aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:59.958704 18491 raft_consensus.cc:729] T 8942dea468384e7d83ff1a7890118deb P d5b10d16e38945e287215c52aa4c9aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5b10d16e38945e287215c52aa4c9aca, State: Initialized, Role: FOLLOWER
I20260928 22:48:59.958752 18491 consensus_queue.cc:260] T 8942dea468384e7d83ff1a7890118deb P d5b10d16e38945e287215c52aa4c9aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19bcd4777c748638a4a12129d17face" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41783 } } peers { permanent_uuid: "f168b2278558423e9b6d11ecc9314584" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32883 } } peers { permanent_uuid: "d5b10d16e38945e287215c52aa4c9aca" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34453 } }
I20260928 22:48:59.958840 18491 ts_tablet_manager.cc:1428] T 8942dea468384e7d83ff1a7890118deb P d5b10d16e38945e287215c52aa4c9aca: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:59.958932 18509 tablet_replica.cc:439] T 8942dea468384e7d83ff1a7890118deb P d5b10d16e38945e287215c52aa4c9aca: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19bcd4777c748638a4a12129d17face" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41783 } } peers { permanent_uuid: "f168b2278558423e9b6d11ecc9314584" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32883 } } peers { permanent_uuid: "d5b10d16e38945e287215c52aa4c9aca" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34453 } } }
I20260928 22:48:59.959000 18509 tablet_replica.cc:442] T 8942dea468384e7d83ff1a7890118deb P d5b10d16e38945e287215c52aa4c9aca: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:48:59.959510 18492 tablet_bootstrap.cc:492] T 8942dea468384e7d83ff1a7890118deb P f168b2278558423e9b6d11ecc9314584: No bootstrap required, opened a new log
I20260928 22:48:59.959564 18492 ts_tablet_manager.cc:1397] T 8942dea468384e7d83ff1a7890118deb P f168b2278558423e9b6d11ecc9314584: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 22:48:59.959729 18492 raft_consensus.cc:348] T 8942dea468384e7d83ff1a7890118deb P f168b2278558423e9b6d11ecc9314584 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19bcd4777c748638a4a12129d17face" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41783 } } peers { permanent_uuid: "f168b2278558423e9b6d11ecc9314584" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32883 } } peers { permanent_uuid: "d5b10d16e38945e287215c52aa4c9aca" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34453 } }
I20260928 22:48:59.959792 18492 raft_consensus.cc:374] T 8942dea468384e7d83ff1a7890118deb P f168b2278558423e9b6d11ecc9314584 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:48:59.959806 18492 raft_consensus.cc:729] T 8942dea468384e7d83ff1a7890118deb P f168b2278558423e9b6d11ecc9314584 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f168b2278558423e9b6d11ecc9314584, State: Initialized, Role: FOLLOWER
I20260928 22:48:59.959839 18492 consensus_queue.cc:260] T 8942dea468384e7d83ff1a7890118deb P f168b2278558423e9b6d11ecc9314584 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19bcd4777c748638a4a12129d17face" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41783 } } peers { permanent_uuid: "f168b2278558423e9b6d11ecc9314584" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32883 } } peers { permanent_uuid: "d5b10d16e38945e287215c52aa4c9aca" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34453 } }
I20260928 22:48:59.959928 18492 ts_tablet_manager.cc:1428] T 8942dea468384e7d83ff1a7890118deb P f168b2278558423e9b6d11ecc9314584: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:48:59.959983 18497 tablet_replica.cc:439] T 8942dea468384e7d83ff1a7890118deb P f168b2278558423e9b6d11ecc9314584: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19bcd4777c748638a4a12129d17face" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41783 } } peers { permanent_uuid: "f168b2278558423e9b6d11ecc9314584" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32883 } } peers { permanent_uuid: "d5b10d16e38945e287215c52aa4c9aca" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34453 } } }
I20260928 22:48:59.960058 18497 tablet_replica.cc:442] T 8942dea468384e7d83ff1a7890118deb P f168b2278558423e9b6d11ecc9314584: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:49:00.131716 18497 raft_consensus.cc:482] T 8942dea468384e7d83ff1a7890118deb P f168b2278558423e9b6d11ecc9314584 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:49:00.131832 18497 raft_consensus.cc:504] T 8942dea468384e7d83ff1a7890118deb P f168b2278558423e9b6d11ecc9314584 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19bcd4777c748638a4a12129d17face" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41783 } } peers { permanent_uuid: "f168b2278558423e9b6d11ecc9314584" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32883 } } peers { permanent_uuid: "d5b10d16e38945e287215c52aa4c9aca" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34453 } }
I20260928 22:49:00.132273 18497 leader_election.cc:288] T 8942dea468384e7d83ff1a7890118deb P f168b2278558423e9b6d11ecc9314584 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c19bcd4777c748638a4a12129d17face (127.16.121.129:41783), d5b10d16e38945e287215c52aa4c9aca (127.16.121.131:34453)
I20260928 22:49:00.135635 18488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:00.132640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:42573 (local address 127.16.121.129:41783)
0928 22:49:00.132707 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:00.132710 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:00.132802 (+ 92us) server_negotiation.cc:407] Connection header received
0928 22:49:00.132931 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:00.132932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:00.132952 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:00.133016 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:49:00.133281 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:00.133777 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:00.134796 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:00.135271 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:00.135331 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:00.135338 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:00.135501 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 22:49:00.135560 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 22:49:00.135687 18489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:00.132536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:49791 (local address 127.16.121.131:34453)
0928 22:49:00.132595 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:00.132597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:00.132758 (+ 161us) server_negotiation.cc:407] Connection header received
0928 22:49:00.132816 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:00.132818 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:00.132832 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:00.132892 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:49:00.133145 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:00.133759 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:00.134831 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:00.135324 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:00.135408 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:00.135419 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:00.135571 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 22:49:00.135623 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 22:49:00.135689 18532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:00.132547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:41783 (local address 127.16.121.130:42573)
0928 22:49:00.132683 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:00.132695 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:00.132810 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:00.133046 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:00.133088 (+ 42us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:00.133091 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:49:00.133196 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:00.133198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:00.133941 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:00.133942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:00.134705 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:00.134707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:00.135347 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:00.135348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:00.135425 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:00.135430 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:49:00.135483 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 22:49:00.135520 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":71,"threads_started":1}
I20260928 22:49:00.135694 18531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:00.132477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.131:34453 (local address 127.16.121.130:49791)
0928 22:49:00.132630 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:00.132645 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:00.132732 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:00.132949 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:00.132950 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:00.132953 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:49:00.133018 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:00.133020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:00.133845 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:00.133846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:00.134710 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:00.134712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:00.135404 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:00.135405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:00.135490 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:00.135495 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:49:00.135552 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 22:49:00.135593 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":69,"threads_started":1}
I20260928 22:49:00.135946 18142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8942dea468384e7d83ff1a7890118deb" candidate_uuid: "f168b2278558423e9b6d11ecc9314584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c19bcd4777c748638a4a12129d17face" is_pre_election: true
I20260928 22:49:00.136062 18142 raft_consensus.cc:2454] T 8942dea468384e7d83ff1a7890118deb P c19bcd4777c748638a4a12129d17face [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f168b2278558423e9b6d11ecc9314584 in term 0.
I20260928 22:49:00.136003 18422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8942dea468384e7d83ff1a7890118deb" candidate_uuid: "f168b2278558423e9b6d11ecc9314584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b10d16e38945e287215c52aa4c9aca" is_pre_election: true
I20260928 22:49:00.136173 18422 raft_consensus.cc:2454] T 8942dea468384e7d83ff1a7890118deb P d5b10d16e38945e287215c52aa4c9aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f168b2278558423e9b6d11ecc9314584 in term 0.
I20260928 22:49:00.136274 18216 leader_election.cc:302] T 8942dea468384e7d83ff1a7890118deb P f168b2278558423e9b6d11ecc9314584 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c19bcd4777c748638a4a12129d17face, f168b2278558423e9b6d11ecc9314584; no voters:
I20260928 22:49:00.136443 18497 raft_consensus.cc:2781] T 8942dea468384e7d83ff1a7890118deb P f168b2278558423e9b6d11ecc9314584 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:49:00.136495 18497 raft_consensus.cc:482] T 8942dea468384e7d83ff1a7890118deb P f168b2278558423e9b6d11ecc9314584 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:49:00.136519 18497 raft_consensus.cc:3037] T 8942dea468384e7d83ff1a7890118deb P f168b2278558423e9b6d11ecc9314584 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:00.137271 18497 raft_consensus.cc:504] T 8942dea468384e7d83ff1a7890118deb P f168b2278558423e9b6d11ecc9314584 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19bcd4777c748638a4a12129d17face" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41783 } } peers { permanent_uuid: "f168b2278558423e9b6d11ecc9314584" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32883 } } peers { permanent_uuid: "d5b10d16e38945e287215c52aa4c9aca" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34453 } }
I20260928 22:49:00.137498 18497 leader_election.cc:288] T 8942dea468384e7d83ff1a7890118deb P f168b2278558423e9b6d11ecc9314584 [CANDIDATE]: Term 1 election: Requested vote from peers c19bcd4777c748638a4a12129d17face (127.16.121.129:41783), d5b10d16e38945e287215c52aa4c9aca (127.16.121.131:34453)
I20260928 22:49:00.137679 18142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8942dea468384e7d83ff1a7890118deb" candidate_uuid: "f168b2278558423e9b6d11ecc9314584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c19bcd4777c748638a4a12129d17face"
I20260928 22:49:00.137694 18422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8942dea468384e7d83ff1a7890118deb" candidate_uuid: "f168b2278558423e9b6d11ecc9314584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b10d16e38945e287215c52aa4c9aca"
I20260928 22:49:00.137758 18142 raft_consensus.cc:3037] T 8942dea468384e7d83ff1a7890118deb P c19bcd4777c748638a4a12129d17face [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:00.137782 18422 raft_consensus.cc:3037] T 8942dea468384e7d83ff1a7890118deb P d5b10d16e38945e287215c52aa4c9aca [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:00.138366 18142 raft_consensus.cc:2454] T 8942dea468384e7d83ff1a7890118deb P c19bcd4777c748638a4a12129d17face [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f168b2278558423e9b6d11ecc9314584 in term 1.
I20260928 22:49:00.138545 18216 leader_election.cc:302] T 8942dea468384e7d83ff1a7890118deb P f168b2278558423e9b6d11ecc9314584 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c19bcd4777c748638a4a12129d17face, f168b2278558423e9b6d11ecc9314584; no voters:
I20260928 22:49:00.138554 18422 raft_consensus.cc:2454] T 8942dea468384e7d83ff1a7890118deb P d5b10d16e38945e287215c52aa4c9aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f168b2278558423e9b6d11ecc9314584 in term 1.
I20260928 22:49:00.138778 18497 raft_consensus.cc:2781] T 8942dea468384e7d83ff1a7890118deb P f168b2278558423e9b6d11ecc9314584 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:49:00.138927 18497 raft_consensus.cc:686] T 8942dea468384e7d83ff1a7890118deb P f168b2278558423e9b6d11ecc9314584 [term 1 LEADER]: Becoming Leader. State: Replica: f168b2278558423e9b6d11ecc9314584, State: Running, Role: LEADER
I20260928 22:49:00.139003 18497 consensus_queue.cc:237] T 8942dea468384e7d83ff1a7890118deb P f168b2278558423e9b6d11ecc9314584 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19bcd4777c748638a4a12129d17face" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41783 } } peers { permanent_uuid: "f168b2278558423e9b6d11ecc9314584" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32883 } } peers { permanent_uuid: "d5b10d16e38945e287215c52aa4c9aca" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34453 } }
I20260928 22:49:00.139683 18533 tablet_replica.cc:439] T 8942dea468384e7d83ff1a7890118deb P f168b2278558423e9b6d11ecc9314584: TxnStatusTablet state changed. Reason: New leader f168b2278558423e9b6d11ecc9314584. Latest consensus state: current_term: 1 leader_uuid: "f168b2278558423e9b6d11ecc9314584" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19bcd4777c748638a4a12129d17face" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41783 } } peers { permanent_uuid: "f168b2278558423e9b6d11ecc9314584" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32883 } } peers { permanent_uuid: "d5b10d16e38945e287215c52aa4c9aca" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34453 } } }
I20260928 22:49:00.139784 18533 tablet_replica.cc:442] T 8942dea468384e7d83ff1a7890118deb P f168b2278558423e9b6d11ecc9314584: This TxnStatusTablet replica's current role is: LEADER
I20260928 22:49:00.139922 17985 catalog_manager.cc:5495] T 8942dea468384e7d83ff1a7890118deb P f168b2278558423e9b6d11ecc9314584 reported cstate change: term changed from 0 to 1, leader changed from <none> to f168b2278558423e9b6d11ecc9314584 (127.16.121.130). New cstate: current_term: 1 leader_uuid: "f168b2278558423e9b6d11ecc9314584" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19bcd4777c748638a4a12129d17face" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f168b2278558423e9b6d11ecc9314584" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5b10d16e38945e287215c52aa4c9aca" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34453 } health_report { overall_health: UNKNOWN } } }
I20260928 22:49:00.139993 18535 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 22:49:00.140117 18535 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 22:49:00.190335 18538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:00.187164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37492 (local address 127.16.121.130:32883)
0928 22:49:00.187434 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:00.187437 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:00.187446 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:49:00.187494 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:00.187495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:00.187521 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:00.187628 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:49:00.187808 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:00.188353 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:00.189513 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:00.189740 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:00.189802 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:00.190013 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:49:00.190119 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:49:00.190173 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:00.190245 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 22:49:00.190264 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":53,"threads_started":1}
I20260928 22:49:00.190372 18526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:00.187135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.130:32883 (local address 127.0.0.1:37492)
0928 22:49:00.187230 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:00.187243 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:00.187295 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:00.187637 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:00.187638 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:00.187640 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 22:49:00.187723 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:00.187726 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:00.188432 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:00.188433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:00.189372 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:00.189375 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:00.189815 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:00.189816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:00.189886 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:00.189903 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 22:49:00.190176 (+ 273us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 22:49:00.190184 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 22:49:00.190242 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 22:49:00.190295 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 22:49:00.191813 18142 raft_consensus.cc:1261] T 8942dea468384e7d83ff1a7890118deb P c19bcd4777c748638a4a12129d17face [term 1 FOLLOWER]: Refusing update from remote peer f168b2278558423e9b6d11ecc9314584: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:49:00.191928 18422 raft_consensus.cc:1261] T 8942dea468384e7d83ff1a7890118deb P d5b10d16e38945e287215c52aa4c9aca [term 1 FOLLOWER]: Refusing update from remote peer f168b2278558423e9b6d11ecc9314584: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:49:00.192163 18533 consensus_queue.cc:1035] T 8942dea468384e7d83ff1a7890118deb P f168b2278558423e9b6d11ecc9314584 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c19bcd4777c748638a4a12129d17face" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:49:00.192327 18497 consensus_queue.cc:1035] T 8942dea468384e7d83ff1a7890118deb P f168b2278558423e9b6d11ecc9314584 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b10d16e38945e287215c52aa4c9aca" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:49:00.192879 18499 tablet_replica.cc:439] T 8942dea468384e7d83ff1a7890118deb P c19bcd4777c748638a4a12129d17face: TxnStatusTablet state changed. Reason: New leader f168b2278558423e9b6d11ecc9314584. Latest consensus state: current_term: 1 leader_uuid: "f168b2278558423e9b6d11ecc9314584" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19bcd4777c748638a4a12129d17face" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41783 } } peers { permanent_uuid: "f168b2278558423e9b6d11ecc9314584" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32883 } } peers { permanent_uuid: "d5b10d16e38945e287215c52aa4c9aca" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34453 } } }
I20260928 22:49:00.192976 18499 tablet_replica.cc:442] T 8942dea468384e7d83ff1a7890118deb P c19bcd4777c748638a4a12129d17face: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:49:00.193048 18509 tablet_replica.cc:439] T 8942dea468384e7d83ff1a7890118deb P d5b10d16e38945e287215c52aa4c9aca: TxnStatusTablet state changed. Reason: New leader f168b2278558423e9b6d11ecc9314584. Latest consensus state: current_term: 1 leader_uuid: "f168b2278558423e9b6d11ecc9314584" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19bcd4777c748638a4a12129d17face" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41783 } } peers { permanent_uuid: "f168b2278558423e9b6d11ecc9314584" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32883 } } peers { permanent_uuid: "d5b10d16e38945e287215c52aa4c9aca" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34453 } } }
I20260928 22:49:00.193132 18509 tablet_replica.cc:442] T 8942dea468384e7d83ff1a7890118deb P d5b10d16e38945e287215c52aa4c9aca: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:49:00.194159 18499 tablet_replica.cc:439] T 8942dea468384e7d83ff1a7890118deb P c19bcd4777c748638a4a12129d17face: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f168b2278558423e9b6d11ecc9314584" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19bcd4777c748638a4a12129d17face" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41783 } } peers { permanent_uuid: "f168b2278558423e9b6d11ecc9314584" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32883 } } peers { permanent_uuid: "d5b10d16e38945e287215c52aa4c9aca" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34453 } } }
I20260928 22:49:00.194141 18497 tablet_replica.cc:439] T 8942dea468384e7d83ff1a7890118deb P f168b2278558423e9b6d11ecc9314584: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f168b2278558423e9b6d11ecc9314584" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19bcd4777c748638a4a12129d17face" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41783 } } peers { permanent_uuid: "f168b2278558423e9b6d11ecc9314584" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32883 } } peers { permanent_uuid: "d5b10d16e38945e287215c52aa4c9aca" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34453 } } }
I20260928 22:49:00.194236 18499 tablet_replica.cc:442] T 8942dea468384e7d83ff1a7890118deb P c19bcd4777c748638a4a12129d17face: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:49:00.194238 18497 tablet_replica.cc:442] T 8942dea468384e7d83ff1a7890118deb P f168b2278558423e9b6d11ecc9314584: This TxnStatusTablet replica's current role is: LEADER
I20260928 22:49:00.194468 18497 tablet_replica.cc:439] T 8942dea468384e7d83ff1a7890118deb P f168b2278558423e9b6d11ecc9314584: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f168b2278558423e9b6d11ecc9314584" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19bcd4777c748638a4a12129d17face" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41783 } } peers { permanent_uuid: "f168b2278558423e9b6d11ecc9314584" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32883 } } peers { permanent_uuid: "d5b10d16e38945e287215c52aa4c9aca" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34453 } } }
I20260928 22:49:00.194551 18497 tablet_replica.cc:442] T 8942dea468384e7d83ff1a7890118deb P f168b2278558423e9b6d11ecc9314584: This TxnStatusTablet replica's current role is: LEADER
I20260928 22:49:00.195266 18509 tablet_replica.cc:439] T 8942dea468384e7d83ff1a7890118deb P d5b10d16e38945e287215c52aa4c9aca: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f168b2278558423e9b6d11ecc9314584" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19bcd4777c748638a4a12129d17face" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41783 } } peers { permanent_uuid: "f168b2278558423e9b6d11ecc9314584" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32883 } } peers { permanent_uuid: "d5b10d16e38945e287215c52aa4c9aca" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34453 } } }
I20260928 22:49:00.195364 18509 tablet_replica.cc:442] T 8942dea468384e7d83ff1a7890118deb P d5b10d16e38945e287215c52aa4c9aca: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:49:00.201496 18538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:00.198717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:39343 (local address 127.16.121.130:32883)
0928 22:49:00.198765 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:00.198767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:00.198851 (+ 84us) server_negotiation.cc:407] Connection header received
0928 22:49:00.198939 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:00.198939 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:00.198948 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:00.198994 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:49:00.199134 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:00.199599 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:00.200691 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:00.200949 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:00.201052 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:00.201243 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:49:00.201270 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:49:00.201309 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:00.201423 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 22:49:00.201443 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 22:49:00.201612 18550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:00.198637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.130:32883 (local address 127.16.121.131:39343)
0928 22:49:00.198790 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:00.198803 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:00.198868 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:00.199005 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:00.199006 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:00.199008 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 22:49:00.199083 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:00.199084 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:00.199675 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:00.199677 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:00.200578 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:00.200581 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:00.201071 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:00.201073 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:00.201137 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:00.201152 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 22:49:00.201329 (+ 177us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 22:49:00.201331 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 22:49:00.201471 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 22:49:00.201530 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":75,"threads_started":1}
I20260928 22:49:00.204121 18282 raft_consensus.cc:1261] T 00ab988c95f644ba8defff5ed10dd6c9 P f168b2278558423e9b6d11ecc9314584 [term 1 FOLLOWER]: Refusing update from remote peer d5b10d16e38945e287215c52aa4c9aca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:49:00.204141 18142 raft_consensus.cc:1261] T 00ab988c95f644ba8defff5ed10dd6c9 P c19bcd4777c748638a4a12129d17face [term 1 FOLLOWER]: Refusing update from remote peer d5b10d16e38945e287215c52aa4c9aca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:49:00.204355 18509 consensus_queue.cc:1035] T 00ab988c95f644ba8defff5ed10dd6c9 P d5b10d16e38945e287215c52aa4c9aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "f168b2278558423e9b6d11ecc9314584" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:49:00.204478 18501 consensus_queue.cc:1035] T 00ab988c95f644ba8defff5ed10dd6c9 P d5b10d16e38945e287215c52aa4c9aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "c19bcd4777c748638a4a12129d17face" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:49:00.213742 18489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:00.210588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:42669 (local address 127.16.121.131:34453)
0928 22:49:00.210627 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:00.210629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:00.210892 (+ 263us) server_negotiation.cc:407] Connection header received
0928 22:49:00.210976 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:00.210977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:00.210989 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:00.211039 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:49:00.211324 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:00.211854 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:00.212945 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:00.213228 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:00.213275 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:00.213507 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:49:00.213537 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:49:00.213583 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:00.213665 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 22:49:00.213685 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:49:00.213829 18554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:00.210526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.131:34453 (local address 127.16.121.130:42669)
0928 22:49:00.210828 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:00.210840 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:00.210906 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:00.211208 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:00.211208 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:00.211210 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 22:49:00.211261 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:00.211262 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:00.211950 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:00.211951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:00.212817 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:00.212820 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:00.213354 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:00.213355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:00.213408 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:00.213424 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 22:49:00.213603 (+ 179us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 22:49:00.213606 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:49:00.213663 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 22:49:00.213712 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":85,"threads_started":1}
I20260928 22:49:00.226787 17984 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:49:00.226904 17984 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:49:00.227645 17984 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 545769a425e048d9a2cb5f05d46f2f68: Sending DeleteTablet for 3 replicas of tablet 00ab988c95f644ba8defff5ed10dd6c9
I20260928 22:49:00.227891 18122 tablet_service.cc:1514] Processing DeleteTablet for tablet 00ab988c95f644ba8defff5ed10dd6c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:49:00 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:46982
I20260928 22:49:00.228092 18402 tablet_service.cc:1514] Processing DeleteTablet for tablet 00ab988c95f644ba8defff5ed10dd6c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:49:00 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:48176
I20260928 22:49:00.228116 18262 tablet_service.cc:1514] Processing DeleteTablet for tablet 00ab988c95f644ba8defff5ed10dd6c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:49:00 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:37480
I20260928 22:49:00.228350 18556 tablet_replica.cc:330] T 00ab988c95f644ba8defff5ed10dd6c9 P c19bcd4777c748638a4a12129d17face: stopping tablet replica
I20260928 22:49:00.228416 18556 raft_consensus.cc:2229] T 00ab988c95f644ba8defff5ed10dd6c9 P c19bcd4777c748638a4a12129d17face [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:49:00.228469 18556 raft_consensus.cc:2258] T 00ab988c95f644ba8defff5ed10dd6c9 P c19bcd4777c748638a4a12129d17face [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:49:00.228574 18557 tablet_replica.cc:330] T 00ab988c95f644ba8defff5ed10dd6c9 P d5b10d16e38945e287215c52aa4c9aca: stopping tablet replica
I20260928 22:49:00.228587 18558 tablet_replica.cc:330] T 00ab988c95f644ba8defff5ed10dd6c9 P f168b2278558423e9b6d11ecc9314584: stopping tablet replica
I20260928 22:49:00.228641 18558 raft_consensus.cc:2229] T 00ab988c95f644ba8defff5ed10dd6c9 P f168b2278558423e9b6d11ecc9314584 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:49:00.228663 18557 raft_consensus.cc:2229] T 00ab988c95f644ba8defff5ed10dd6c9 P d5b10d16e38945e287215c52aa4c9aca [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:49:00.228698 18558 raft_consensus.cc:2258] T 00ab988c95f644ba8defff5ed10dd6c9 P f168b2278558423e9b6d11ecc9314584 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:49:00.228736 18557 raft_consensus.cc:2258] T 00ab988c95f644ba8defff5ed10dd6c9 P d5b10d16e38945e287215c52aa4c9aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:49:00.228947 18556 ts_tablet_manager.cc:1905] T 00ab988c95f644ba8defff5ed10dd6c9 P c19bcd4777c748638a4a12129d17face: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:49:00.228992 18557 ts_tablet_manager.cc:1905] T 00ab988c95f644ba8defff5ed10dd6c9 P d5b10d16e38945e287215c52aa4c9aca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:49:00.230093 18558 ts_tablet_manager.cc:1905] T 00ab988c95f644ba8defff5ed10dd6c9 P f168b2278558423e9b6d11ecc9314584: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:49:00.230625 18557 ts_tablet_manager.cc:1918] T 00ab988c95f644ba8defff5ed10dd6c9 P d5b10d16e38945e287215c52aa4c9aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 22:49:00.230700 18557 log.cc:1196] T 00ab988c95f644ba8defff5ed10dd6c9 P d5b10d16e38945e287215c52aa4c9aca: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-2/wal/wals/00ab988c95f644ba8defff5ed10dd6c9
I20260928 22:49:00.230947 18557 ts_tablet_manager.cc:1939] T 00ab988c95f644ba8defff5ed10dd6c9 P d5b10d16e38945e287215c52aa4c9aca: Deleting consensus metadata
I20260928 22:49:00.231307 17970 catalog_manager.cc:4841] TS d5b10d16e38945e287215c52aa4c9aca (127.16.121.131:34453): tablet 00ab988c95f644ba8defff5ed10dd6c9 (table test-table [id=66798ae363df4c2c9322901e2209a01c]) successfully deleted
I20260928 22:49:00.231828 18558 ts_tablet_manager.cc:1918] T 00ab988c95f644ba8defff5ed10dd6c9 P f168b2278558423e9b6d11ecc9314584: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 22:49:00.231899 18558 log.cc:1196] T 00ab988c95f644ba8defff5ed10dd6c9 P f168b2278558423e9b6d11ecc9314584: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-1/wal/wals/00ab988c95f644ba8defff5ed10dd6c9
I20260928 22:49:00.231928 18556 ts_tablet_manager.cc:1918] T 00ab988c95f644ba8defff5ed10dd6c9 P c19bcd4777c748638a4a12129d17face: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 22:49:00.231990 18556 log.cc:1196] T 00ab988c95f644ba8defff5ed10dd6c9 P c19bcd4777c748638a4a12129d17face: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790635733329745-16870-0/minicluster-data/ts-0/wal/wals/00ab988c95f644ba8defff5ed10dd6c9
I20260928 22:49:00.232155 18558 ts_tablet_manager.cc:1939] T 00ab988c95f644ba8defff5ed10dd6c9 P f168b2278558423e9b6d11ecc9314584: Deleting consensus metadata
I20260928 22:49:00.232278 18556 ts_tablet_manager.cc:1939] T 00ab988c95f644ba8defff5ed10dd6c9 P c19bcd4777c748638a4a12129d17face: Deleting consensus metadata
I20260928 22:49:00.232487 17969 catalog_manager.cc:4841] TS f168b2278558423e9b6d11ecc9314584 (127.16.121.130:32883): tablet 00ab988c95f644ba8defff5ed10dd6c9 (table test-table [id=66798ae363df4c2c9322901e2209a01c]) successfully deleted
I20260928 22:49:00.232630 17972 catalog_manager.cc:4841] TS c19bcd4777c748638a4a12129d17face (127.16.121.129:41783): tablet 00ab988c95f644ba8defff5ed10dd6c9 (table test-table [id=66798ae363df4c2c9322901e2209a01c]) successfully deleted
Sep 28 22:49:00 dist-test-slave-56vw krb5kdc[17940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635739, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.121.129@KRBTEST.COM
I20260928 22:49:00.241987 18488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:00.232866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46998 (local address 127.16.121.129:41783)
0928 22:49:00.232914 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:00.232916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:00.233226 (+ 310us) server_negotiation.cc:407] Connection header received
0928 22:49:00.233256 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:00.233257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:00.233270 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:00.233334 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:00.234002 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:00.234471 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:00.235493 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:00.235695 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:00.235752 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:00.238597 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:00.238627 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:00.238627 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:00.238636 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:00.240697 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:00.241087 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:00.241091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:00.241091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:00.241176 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:00.241369 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:00.241370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:00.241370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:00.241560 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:00.241727 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:00.241786 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 22:49:00.241874 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 22:49:00.242348 18182 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46998
I20260928 22:49:00.250008 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:00.243408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40768 (local address 127.16.121.190:33833)
0928 22:49:00.243457 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:00.243459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:00.243516 (+ 57us) server_negotiation.cc:407] Connection header received
0928 22:49:00.243578 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:00.243579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:00.243593 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:00.243664 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:00.244124 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:00.244887 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:00.245576 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:00.245891 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:00.245985 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:00.247273 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:00.247274 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:00.247275 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:00.247282 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:00.248849 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:00.249185 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:00.249186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:00.249187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:00.249253 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:00.249436 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:00.249437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:00.249437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:00.249585 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:00.249691 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:00.249802 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 22:49:00.249896 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260928 22:49:00.250228 17984 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40768
I20260928 22:49:00.250739 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 18057
I20260928 22:49:00.256716 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 18197
I20260928 22:49:00.262571 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 18337
I20260928 22:49:00.268291 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 17953
2026-09-28T22:49:00Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2740 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:49:00 dist-test-slave-56vw krb5kdc[18565](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:49:00 dist-test-slave-56vw krb5kdc[18565](info): set up 2 sockets
Sep 28 22:49:00 dist-test-slave-56vw krb5kdc[18565](info): commencing operation
krb5kdc: starting...
W20260928 22:49:01.323196 16870 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Sep 28 22:49:01 dist-test-slave-56vw krb5kdc[18565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:49:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:49:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.121.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
I20260928 22:49:01.456197 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:40849
--webserver_interface=127.16.121.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:33309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:40849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:49:01.544232 18578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:01.544481 18578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:01.544508 18578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:01.545774 18578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:49:01.545835 18578 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 22:49:01.545849 18578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:01.545862 18578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:01.545876 18578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:49:01.545887 18578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:49:01.547583 18578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:33309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:40849
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:40849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.121.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:01.547880 18578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:01.548166 18578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:01.553843 18584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:01.553891 18586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:01.553874 18583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:01.553933 18578 server_base.cc:1034] running on GCE node
I20260928 22:49:01.554299 18578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:01.554656 18578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:01.555840 18578 hybrid_clock.cc:648] HybridClock initialized: now 1790635741555827 us; error 31 us; skew 500 ppm
Sep 28 22:49:01 dist-test-slave-56vw krb5kdc[18565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635741, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:01.558638 18578 init.cc:377] Logged in from keytab as kudu/127.16.121.190@KRBTEST.COM (short username kudu)
I20260928 22:49:01.559099 18578 webserver.cc:460] Webserver started at http://127.16.121.190:40227/ using document root <none> and password file <none>
I20260928 22:49:01.559320 18578 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:01.559363 18578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:01.559463 18578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:01.560617 18578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "622784bfe8d5446abd97cdd40c6d8115"
format_stamp: "Formatted at 2026-09-28 22:49:01 on dist-test-slave-56vw"
server_key: "c4b5093810bb74ed37e6fa090d81be63"
server_key_iv: "ac89214d32f86b0395bfc4630d48593f"
server_key_version: "encryptionkey@0"
I20260928 22:49:01.561167 18578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "622784bfe8d5446abd97cdd40c6d8115"
format_stamp: "Formatted at 2026-09-28 22:49:01 on dist-test-slave-56vw"
server_key: "c4b5093810bb74ed37e6fa090d81be63"
server_key_iv: "ac89214d32f86b0395bfc4630d48593f"
server_key_version: "encryptionkey@0"
I20260928 22:49:01.562903 18578 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:49:01.563966 18593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:01.564198 18578 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:01.564288 18578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "622784bfe8d5446abd97cdd40c6d8115"
format_stamp: "Formatted at 2026-09-28 22:49:01 on dist-test-slave-56vw"
server_key: "c4b5093810bb74ed37e6fa090d81be63"
server_key_iv: "ac89214d32f86b0395bfc4630d48593f"
server_key_version: "encryptionkey@0"
I20260928 22:49:01.564354 18578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:01.593798 18578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:01.596628 18578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:01.596884 18578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:01.603227 18578 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.190:40849
I20260928 22:49:01.603361 18655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.190:40849 every 8 connection(s)
I20260928 22:49:01.603713 18578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:49:01.604550 18656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:01.607329 18656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 622784bfe8d5446abd97cdd40c6d8115: Bootstrap starting.
I20260928 22:49:01.608079 18656 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 622784bfe8d5446abd97cdd40c6d8115: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:01.608414 18656 log.cc:824] T 00000000000000000000000000000000 P 622784bfe8d5446abd97cdd40c6d8115: Log is configured to *not* fsync() on all Append() calls
I20260928 22:49:01.609259 18656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 622784bfe8d5446abd97cdd40c6d8115: No bootstrap required, opened a new log
I20260928 22:49:01.611572 18656 raft_consensus.cc:348] T 00000000000000000000000000000000 P 622784bfe8d5446abd97cdd40c6d8115 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "622784bfe8d5446abd97cdd40c6d8115" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 40849 } }
I20260928 22:49:01.611740 18656 raft_consensus.cc:374] T 00000000000000000000000000000000 P 622784bfe8d5446abd97cdd40c6d8115 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:01.611764 18656 raft_consensus.cc:729] T 00000000000000000000000000000000 P 622784bfe8d5446abd97cdd40c6d8115 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 622784bfe8d5446abd97cdd40c6d8115, State: Initialized, Role: FOLLOWER
I20260928 22:49:01.611931 18656 consensus_queue.cc:260] T 00000000000000000000000000000000 P 622784bfe8d5446abd97cdd40c6d8115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "622784bfe8d5446abd97cdd40c6d8115" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 40849 } }
I20260928 22:49:01.612020 18656 raft_consensus.cc:388] T 00000000000000000000000000000000 P 622784bfe8d5446abd97cdd40c6d8115 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:49:01.612044 18656 raft_consensus.cc:482] T 00000000000000000000000000000000 P 622784bfe8d5446abd97cdd40c6d8115 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:49:01.612099 18656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 622784bfe8d5446abd97cdd40c6d8115 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:01.612707 18656 raft_consensus.cc:504] T 00000000000000000000000000000000 P 622784bfe8d5446abd97cdd40c6d8115 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "622784bfe8d5446abd97cdd40c6d8115" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 40849 } }
I20260928 22:49:01.612843 18656 leader_election.cc:302] T 00000000000000000000000000000000 P 622784bfe8d5446abd97cdd40c6d8115 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 622784bfe8d5446abd97cdd40c6d8115; no voters:
I20260928 22:49:01.613009 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 18578
I20260928 22:49:01.613055 18656 leader_election.cc:288] T 00000000000000000000000000000000 P 622784bfe8d5446abd97cdd40c6d8115 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:49:01.613097 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:49:01.613149 18659 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 622784bfe8d5446abd97cdd40c6d8115 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:49:01.613307 16870 external_mini_cluster.cc:1428] Setting key ee9f23123a915ec71dccd02327ab9449
I20260928 22:49:01.613323 18659 raft_consensus.cc:686] T 00000000000000000000000000000000 P 622784bfe8d5446abd97cdd40c6d8115 [term 1 LEADER]: Becoming Leader. State: Replica: 622784bfe8d5446abd97cdd40c6d8115, State: Running, Role: LEADER
I20260928 22:49:01.613446 18656 sys_catalog.cc:564] T 00000000000000000000000000000000 P 622784bfe8d5446abd97cdd40c6d8115 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:49:01.613489 18659 consensus_queue.cc:237] T 00000000000000000000000000000000 P 622784bfe8d5446abd97cdd40c6d8115 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "622784bfe8d5446abd97cdd40c6d8115" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 40849 } }
I20260928 22:49:01.613984 18660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 622784bfe8d5446abd97cdd40c6d8115 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "622784bfe8d5446abd97cdd40c6d8115" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "622784bfe8d5446abd97cdd40c6d8115" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 40849 } } }
I20260928 22:49:01.614061 18660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 622784bfe8d5446abd97cdd40c6d8115 [sys.catalog]: This master's current role is: LEADER
I20260928 22:49:01.614446 18661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 622784bfe8d5446abd97cdd40c6d8115 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 622784bfe8d5446abd97cdd40c6d8115. Latest consensus state: current_term: 1 leader_uuid: "622784bfe8d5446abd97cdd40c6d8115" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "622784bfe8d5446abd97cdd40c6d8115" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 40849 } } }
I20260928 22:49:01.614499 18670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:49:01.614626 18661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 622784bfe8d5446abd97cdd40c6d8115 [sys.catalog]: This master's current role is: LEADER
I20260928 22:49:01.615336 18670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:49:01.617015 18670 catalog_manager.cc:1347] Generated new cluster ID: 60bcfe22443648eb895fc495a981c414
I20260928 22:49:01.617069 18670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 22:49:01 dist-test-slave-56vw krb5kdc[18565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:49:01.626924 18667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:01.613965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54416 (local address 127.16.121.190:40849)
0928 22:49:01.614155 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:01.614167 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:01.614366 (+ 199us) server_negotiation.cc:407] Connection header received
0928 22:49:01.614603 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:01.614613 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:01.614675 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:01.614789 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:01.615823 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:01.617336 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:01.618530 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:01.619285 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:01.619423 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:01.622651 (+ 3228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:01.622657 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:01.622658 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:01.622685 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:01.624987 (+ 2302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:01.625460 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:01.625465 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:01.625465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:01.625619 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:01.625841 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:01.625842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:01.625842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:01.626224 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:01.626390 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:01.626520 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 22:49:01.626670 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":63,"threads_started":1}
I20260928 22:49:01.629776 18670 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:49:01.630342 18670 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:49:01.635191 18670 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 622784bfe8d5446abd97cdd40c6d8115: Generated new TSK 0
I20260928 22:49:01.635540 18670 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.121.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab.
Entry for principal kudu/127.16.121.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab.
I20260928 22:49:01.667769 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.129:0
--local_ip_for_outbound_sockets=127.16.121.129
--webserver_interface=127.16.121.129
--webserver_port=0
--tserver_master_addrs=127.16.121.190:40849
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab
--principal=kudu/127.16.121.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:33309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:49:01.763788 18682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:01.764058 18682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:01.764108 18682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:01.766219 18682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:01.766289 18682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:01.766324 18682 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 22:49:01.766350 18682 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 22:49:01.766377 18682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.129
I20260928 22:49:01.769224 18682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:33309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab
--principal=kudu/127.16.121.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.121.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.121.190:40849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.129
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:01.769595 18682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:01.769918 18682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:01.777365 18688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:01.777536 18682 server_base.cc:1034] running on GCE node
W20260928 22:49:01.777647 18690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:01.777797 18687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:01.778043 18682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:01.778337 18682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:01.779526 18682 hybrid_clock.cc:648] HybridClock initialized: now 1790635741779502 us; error 32 us; skew 500 ppm
Sep 28 22:49:01 dist-test-slave-56vw krb5kdc[18565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635741, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:01.782712 18682 init.cc:377] Logged in from keytab as kudu/127.16.121.129@KRBTEST.COM (short username kudu)
I20260928 22:49:01.783349 18682 webserver.cc:460] Webserver started at http://127.16.121.129:45309/ using document root <none> and password file <none>
I20260928 22:49:01.783664 18682 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:01.783743 18682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:01.783900 18682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:01.785385 18682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-0/data/instance:
uuid: "5e5a0e809d9842da90487a529361bc7d"
format_stamp: "Formatted at 2026-09-28 22:49:01 on dist-test-slave-56vw"
server_key: "e303882ef774fdf487218f8b82b5c219"
server_key_iv: "e3957d84aacd7d2922d072be882203be"
server_key_version: "encryptionkey@0"
I20260928 22:49:01.785836 18682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance:
uuid: "5e5a0e809d9842da90487a529361bc7d"
format_stamp: "Formatted at 2026-09-28 22:49:01 on dist-test-slave-56vw"
server_key: "e303882ef774fdf487218f8b82b5c219"
server_key_iv: "e3957d84aacd7d2922d072be882203be"
server_key_version: "encryptionkey@0"
I20260928 22:49:01.787737 18682 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260928 22:49:01.788869 18697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:01.789203 18682 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:01.789302 18682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-0/wal
uuid: "5e5a0e809d9842da90487a529361bc7d"
format_stamp: "Formatted at 2026-09-28 22:49:01 on dist-test-slave-56vw"
server_key: "e303882ef774fdf487218f8b82b5c219"
server_key_iv: "e3957d84aacd7d2922d072be882203be"
server_key_version: "encryptionkey@0"
I20260928 22:49:01.789398 18682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:01.824508 18682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:01.826694 18682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:01.826820 18682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:01.827416 18682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:49:01.827471 18682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:01.827498 18682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:49:01.827512 18682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:01.835348 18682 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.129:41735
I20260928 22:49:01.835502 18812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.129:41735 every 8 connection(s)
Sep 28 22:49:01 dist-test-slave-56vw krb5kdc[18565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635741, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:49:01.835857 18682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
I20260928 22:49:01.839438 18667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:01.827979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:37733 (local address 127.16.121.190:40849)
0928 22:49:01.828034 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:01.828037 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:01.828237 (+ 200us) server_negotiation.cc:407] Connection header received
0928 22:49:01.828776 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:01.828777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:01.828794 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:01.828884 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:01.829999 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:01.830832 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:01.832787 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:01.833031 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:01.833091 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:01.836355 (+ 3264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:01.836357 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:01.836358 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:01.836368 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:01.837759 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:01.838218 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:01.838219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:01.838220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:01.838284 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:01.838660 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:01.838661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:01.838662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:01.838791 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:01.838862 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:01.839276 (+ 414us) server_negotiation.cc:299] Negotiation successful
0928 22:49:01.839362 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 22:49:01.839612 18706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:01.827954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:40849 (local address 127.16.121.129:37733)
0928 22:49:01.828111 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:01.828163 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:01.828667 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:01.829018 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:01.829024 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:01.829502 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:01.829859 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:01.829867 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:01.830944 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:01.830945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:01.832677 (+ 1732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:01.832681 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:01.833100 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:01.833101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:01.833278 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:01.834099 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:01.834111 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:01.836249 (+ 2138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:01.837881 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:01.837888 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:01.837891 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:01.838121 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:01.838411 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:01.838412 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:01.838413 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:01.838547 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:01.838914 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:01.838921 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:01.839116 (+ 195us) client_negotiation.cc:769] Sending connection context
0928 22:49:01.839233 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 22:49:01.839377 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":92,"threads_started":1}
I20260928 22:49:01.840234 18813 heartbeater.cc:347] Connected to a master server at 127.16.121.190:40849
I20260928 22:49:01.840315 18813 heartbeater.cc:464] Registering TS with master...
I20260928 22:49:01.840523 18813 heartbeater.cc:511] Master 127.16.121.190:40849 requested a full tablet report, sending...
I20260928 22:49:01.841116 18610 ts_manager.cc:194] Registered new tserver with Master: 5e5a0e809d9842da90487a529361bc7d (127.16.121.129:41735)
I20260928 22:49:01.841988 18610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.129@KRBTEST.COM'} at 127.16.121.129:37733
I20260928 22:49:01.844192 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 18682
I20260928 22:49:01.844298 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance
I20260928 22:49:01.844515 16870 external_mini_cluster.cc:1428] Setting key c929a204dd5ed7dead0ba5a1a89fe833
I20260928 22:49:01.848030 18667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:01.841495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:33921 (local address 127.16.121.190:40849)
0928 22:49:01.841543 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:01.841545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:01.841623 (+ 78us) server_negotiation.cc:407] Connection header received
0928 22:49:01.841688 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:01.841689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:01.841698 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:01.841748 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:01.842051 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:01.842722 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:01.843536 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:01.843723 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:01.843792 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:01.845069 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:01.845070 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:01.845071 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:01.845079 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:01.846614 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:01.847090 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:01.847091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:01.847091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:01.847155 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:01.847364 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:01.847364 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:01.847365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:01.847599 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:01.847675 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:01.847879 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 22:49:01.847966 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 22:49:01.848065 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:01.841422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:40849 (local address 127.16.121.129:33921)
0928 22:49:01.841558 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:01.841569 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:01.841632 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:01.841755 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:01.841755 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:01.841917 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:01.841969 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:01.841970 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:01.842829 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:01.842830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:01.843448 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:01.843451 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:01.843835 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:01.843836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:01.843907 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:01.844579 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:01.844585 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:01.844980 (+ 395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:01.846715 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:01.846717 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:01.846717 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:01.846977 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:01.847234 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:01.847235 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:01.847235 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:01.847295 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:01.847684 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:01.847686 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:01.847744 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 22:49:01.847830 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 22:49:01.847989 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":71,"threads_started":1}
WARNING: no policy specified for kudu/127.16.121.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab.
Entry for principal kudu/127.16.121.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab.
I20260928 22:49:01.875211 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.130:0
--local_ip_for_outbound_sockets=127.16.121.130
--webserver_interface=127.16.121.130
--webserver_port=0
--tserver_master_addrs=127.16.121.190:40849
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab
--principal=kudu/127.16.121.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:33309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:49:01.964184 18822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:01.964408 18822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:01.964435 18822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:01.965631 18822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:01.965677 18822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:01.965698 18822 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 22:49:01.965723 18822 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 22:49:01.965740 18822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.130
I20260928 22:49:01.967414 18822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:33309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab
--principal=kudu/127.16.121.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.121.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.121.190:40849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.130
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:01.967814 18822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:01.968082 18822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:01.973543 18828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:01.973428 18827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:01.973556 18822 server_base.cc:1034] running on GCE node
W20260928 22:49:01.973431 18830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:01.973897 18822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:01.974180 18822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:01.975353 18822 hybrid_clock.cc:648] HybridClock initialized: now 1790635741975346 us; error 38 us; skew 500 ppm
Sep 28 22:49:01 dist-test-slave-56vw krb5kdc[18565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635741, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:01.978125 18822 init.cc:377] Logged in from keytab as kudu/127.16.121.130@KRBTEST.COM (short username kudu)
I20260928 22:49:01.978631 18822 webserver.cc:460] Webserver started at http://127.16.121.130:41417/ using document root <none> and password file <none>
I20260928 22:49:01.978874 18822 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:01.978941 18822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:01.979120 18822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:01.980280 18822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-1/data/instance:
uuid: "9d7ed45c903f4384989599e40b950c9f"
format_stamp: "Formatted at 2026-09-28 22:49:01 on dist-test-slave-56vw"
server_key: "1df99764631391c132a863a47f380594"
server_key_iv: "cf50a640de5863f0268669f9d5ca87b3"
server_key_version: "encryptionkey@0"
I20260928 22:49:01.980690 18822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance:
uuid: "9d7ed45c903f4384989599e40b950c9f"
format_stamp: "Formatted at 2026-09-28 22:49:01 on dist-test-slave-56vw"
server_key: "1df99764631391c132a863a47f380594"
server_key_iv: "cf50a640de5863f0268669f9d5ca87b3"
server_key_version: "encryptionkey@0"
I20260928 22:49:01.982121 18822 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:49:01.982988 18837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:01.983202 18822 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:01.983309 18822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-1/wal
uuid: "9d7ed45c903f4384989599e40b950c9f"
format_stamp: "Formatted at 2026-09-28 22:49:01 on dist-test-slave-56vw"
server_key: "1df99764631391c132a863a47f380594"
server_key_iv: "cf50a640de5863f0268669f9d5ca87b3"
server_key_version: "encryptionkey@0"
I20260928 22:49:01.983385 18822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:01.999683 18822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:02.001950 18822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:02.002118 18822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:02.002771 18822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:49:02.002825 18822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:02.002861 18822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:49:02.002889 18822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 22:49:02 dist-test-slave-56vw krb5kdc[18565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635741, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.130@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:49:02.010638 18822 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.130:38211
I20260928 22:49:02.010741 18952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.130:38211 every 8 connection(s)
I20260928 22:49:02.011111 18822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
I20260928 22:49:02.013126 18667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:02.003638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:34739 (local address 127.16.121.190:40849)
0928 22:49:02.003706 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:02.003709 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:02.003721 (+ 12us) server_negotiation.cc:407] Connection header received
0928 22:49:02.004146 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:02.004148 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:02.004160 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:02.004241 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:02.004999 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.005676 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.006760 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.006960 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.007038 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:02.009948 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:02.009951 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:02.009951 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:02.009960 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:02.011798 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:02.012219 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:02.012220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:02.012220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:02.012274 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:02.012490 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:02.012490 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:02.012490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:02.012600 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:02.012650 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:02.012973 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 22:49:02.013061 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 22:49:02.013343 18846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:02.003199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:40849 (local address 127.16.121.130:34739)
0928 22:49:02.003406 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:02.003455 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:02.004041 (+ 586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:02.004388 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:02.004392 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:02.004649 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:02.004840 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:02.004845 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.005784 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.005785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:02.006657 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:02.006659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.007061 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.007063 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:02.007279 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:02.007877 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:02.007891 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:02.009816 (+ 1925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:02.011876 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:02.011880 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:02.011882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:02.012148 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:02.012336 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:02.012336 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:02.012337 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:02.012432 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:02.012656 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:02.012661 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:02.012848 (+ 187us) client_negotiation.cc:769] Sending connection context
0928 22:49:02.012987 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 22:49:02.013119 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":109,"threads_started":1}
I20260928 22:49:02.013859 18953 heartbeater.cc:347] Connected to a master server at 127.16.121.190:40849
I20260928 22:49:02.013973 18953 heartbeater.cc:464] Registering TS with master...
I20260928 22:49:02.014166 18953 heartbeater.cc:511] Master 127.16.121.190:40849 requested a full tablet report, sending...
I20260928 22:49:02.014575 18610 ts_manager.cc:194] Registered new tserver with Master: 9d7ed45c903f4384989599e40b950c9f (127.16.121.130:38211)
I20260928 22:49:02.015138 18610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.130@KRBTEST.COM'} at 127.16.121.130:34739
I20260928 22:49:02.020483 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 18822
I20260928 22:49:02.020660 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance
I20260928 22:49:02.020869 16870 external_mini_cluster.cc:1428] Setting key 37d3bd4e4939bbeb1882498e55122fbe
I20260928 22:49:02.021335 18667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:02.015106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:52505 (local address 127.16.121.190:40849)
0928 22:49:02.015145 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:02.015147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:02.015204 (+ 57us) server_negotiation.cc:407] Connection header received
0928 22:49:02.015264 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:02.015265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:02.015274 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:02.015335 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:02.015768 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.016533 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.017285 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.017495 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.017573 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:02.018538 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:02.018539 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:02.018539 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:02.018546 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:02.020020 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:02.020390 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:02.020391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:02.020392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:02.020447 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:02.020668 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:02.020669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:02.020669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:02.020878 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:02.020958 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:02.021144 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 22:49:02.021240 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 22:49:02.021350 18959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:02.015004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:40849 (local address 127.16.121.130:52505)
0928 22:49:02.015149 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:02.015159 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:02.015217 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:02.015364 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:02.015366 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:02.015621 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:02.015683 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:02.015685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.016656 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.016657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:02.017187 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:02.017189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.017570 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.017571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:02.017635 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:02.018124 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:02.018129 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:02.018423 (+ 294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:02.020126 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:02.020127 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:02.020127 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:02.020304 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:02.020526 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:02.020527 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:02.020527 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:02.020586 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:02.020954 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:02.020955 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:02.021005 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 22:49:02.021115 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 22:49:02.021240 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":81,"threads_started":1}
WARNING: no policy specified for kudu/127.16.121.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab.
Entry for principal kudu/127.16.121.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab.
I20260928 22:49:02.055953 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.131:0
--local_ip_for_outbound_sockets=127.16.121.131
--webserver_interface=127.16.121.131
--webserver_port=0
--tserver_master_addrs=127.16.121.190:40849
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab
--principal=kudu/127.16.121.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:33309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:49:02.144117 18962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:02.144338 18962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:02.144367 18962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:02.145596 18962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:02.145643 18962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:02.145664 18962 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 22:49:02.145676 18962 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 22:49:02.145720 18962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.131
I20260928 22:49:02.147316 18962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:33309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab
--principal=kudu/127.16.121.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.121.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.121.190:40849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.131
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:02.147601 18962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:02.147841 18962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:02.153329 18968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:02.153349 18967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:02.153453 18962 server_base.cc:1034] running on GCE node
W20260928 22:49:02.153329 18970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:02.153777 18962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:02.154036 18962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:02.155186 18962 hybrid_clock.cc:648] HybridClock initialized: now 1790635742155177 us; error 45 us; skew 500 ppm
Sep 28 22:49:02 dist-test-slave-56vw krb5kdc[18565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635742, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:02.157878 18962 init.cc:377] Logged in from keytab as kudu/127.16.121.131@KRBTEST.COM (short username kudu)
I20260928 22:49:02.158326 18962 webserver.cc:460] Webserver started at http://127.16.121.131:42361/ using document root <none> and password file <none>
I20260928 22:49:02.158581 18962 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:02.158648 18962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:02.158802 18962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:02.160040 18962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-2/data/instance:
uuid: "c9a7dce56d024b8998f3a23395492f1c"
format_stamp: "Formatted at 2026-09-28 22:49:02 on dist-test-slave-56vw"
server_key: "c333824c7f8d4924842df1d5d2ce9817"
server_key_iv: "83c780a45ad067a8bc69ec7b7985bf9e"
server_key_version: "encryptionkey@0"
I20260928 22:49:02.160427 18962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance:
uuid: "c9a7dce56d024b8998f3a23395492f1c"
format_stamp: "Formatted at 2026-09-28 22:49:02 on dist-test-slave-56vw"
server_key: "c333824c7f8d4924842df1d5d2ce9817"
server_key_iv: "83c780a45ad067a8bc69ec7b7985bf9e"
server_key_version: "encryptionkey@0"
I20260928 22:49:02.161845 18962 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:49:02.162658 18977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:02.162837 18962 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:02.162935 18962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-2/wal
uuid: "c9a7dce56d024b8998f3a23395492f1c"
format_stamp: "Formatted at 2026-09-28 22:49:02 on dist-test-slave-56vw"
server_key: "c333824c7f8d4924842df1d5d2ce9817"
server_key_iv: "83c780a45ad067a8bc69ec7b7985bf9e"
server_key_version: "encryptionkey@0"
I20260928 22:49:02.163014 18962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:02.184523 18962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:02.187358 18962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:02.187544 18962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:02.188125 18962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:49:02.188198 18962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:02.188249 18962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:49:02.188278 18962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 22:49:02 dist-test-slave-56vw krb5kdc[18565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635742, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.131@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:49:02.196202 18962 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.131:41721
I20260928 22:49:02.196316 19092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.131:41721 every 8 connection(s)
I20260928 22:49:02.196628 18962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
I20260928 22:49:02.199878 18667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:02.188685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:39435 (local address 127.16.121.190:40849)
0928 22:49:02.188762 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:02.188765 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:02.189103 (+ 338us) server_negotiation.cc:407] Connection header received
0928 22:49:02.189696 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:02.189699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:02.189715 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:02.189805 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:02.190809 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.191821 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.192897 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.193292 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.193383 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:02.196791 (+ 3408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:02.196792 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:02.196793 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:02.196808 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:02.198339 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:02.198810 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:02.198811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:02.198811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:02.198882 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:02.199216 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:02.199217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:02.199218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:02.199331 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:02.199395 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:02.199734 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 22:49:02.199810 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 22:49:02.199978 18986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:02.188654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:40849 (local address 127.16.121.131:39435)
0928 22:49:02.188974 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:02.189015 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:02.189549 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:02.189903 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:02.189907 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:02.190290 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:02.190690 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:02.190697 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.191926 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.191928 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:02.192777 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:02.192779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.193407 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.193408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:02.193566 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:02.194674 (+ 1108us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:02.194683 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:02.196643 (+ 1960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:02.198468 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:02.198475 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:02.198476 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:02.198715 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:02.198969 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:02.198969 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:02.198970 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:02.199148 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:02.199393 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:02.199453 (+ 60us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:02.199581 (+ 128us) client_negotiation.cc:769] Sending connection context
0928 22:49:02.199674 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 22:49:02.199801 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":110,"threads_started":1}
I20260928 22:49:02.200572 19093 heartbeater.cc:347] Connected to a master server at 127.16.121.190:40849
I20260928 22:49:02.200675 19093 heartbeater.cc:464] Registering TS with master...
I20260928 22:49:02.200877 19093 heartbeater.cc:511] Master 127.16.121.190:40849 requested a full tablet report, sending...
I20260928 22:49:02.201263 18609 ts_manager.cc:194] Registered new tserver with Master: c9a7dce56d024b8998f3a23395492f1c (127.16.121.131:41721)
I20260928 22:49:02.201803 18609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.131@KRBTEST.COM'} at 127.16.121.131:39435
I20260928 22:49:02.202092 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 18962
I20260928 22:49:02.202176 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance
I20260928 22:49:02.202389 16870 external_mini_cluster.cc:1428] Setting key e919a86655a7630eae07dbfff8e4b23d
I20260928 22:49:02.203172 16870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:49:02.208243 19099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:02.201621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:40849 (local address 127.16.121.131:51055)
0928 22:49:02.201906 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:02.201919 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:02.201990 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:02.202124 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:02.202125 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:02.202422 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:02.202503 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:02.202506 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.203481 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.203481 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:02.203972 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:02.203973 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.204476 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.204477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:02.204548 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:02.205028 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:02.205035 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:02.205449 (+ 414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:02.207190 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:02.207190 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:02.207191 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:02.207359 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:02.207592 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:02.207593 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:02.207593 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:02.207649 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:02.207902 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:02.207903 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:02.207960 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 22:49:02.208034 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 22:49:02.208143 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":167,"threads_started":1}
I20260928 22:49:02.208237 18667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:02.201818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:51055 (local address 127.16.121.190:40849)
0928 22:49:02.201904 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:02.201906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:02.201995 (+ 89us) server_negotiation.cc:407] Connection header received
0928 22:49:02.202050 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:02.202051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:02.202059 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:02.202108 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:02.202608 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.203372 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.204069 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.204381 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.204456 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:02.205575 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:02.205576 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:02.205576 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:02.205582 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:02.207053 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:02.207448 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:02.207449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:02.207449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:02.207506 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:02.207714 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:02.207715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:02.207716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:02.207834 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:02.207911 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:02.208062 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 22:49:02.208141 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Sep 28 22:49:02 dist-test-slave-56vw krb5kdc[18565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 22:49:02 dist-test-slave-56vw krb5kdc[18565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:49:02.229140 18667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:02.220823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54420 (local address 127.16.121.190:40849)
0928 22:49:02.220878 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:02.220880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:02.221004 (+ 124us) server_negotiation.cc:407] Connection header received
0928 22:49:02.221062 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:02.221063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:02.221073 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:02.221126 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:02.221839 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.222386 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.223260 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.223526 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.223589 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:02.226203 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:02.226205 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:02.226205 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:02.226214 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:02.227809 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:02.228190 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:02.228191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:02.228192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:02.228246 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:02.228515 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:02.228516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:02.228517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:02.228634 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:02.228717 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:02.228921 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 22:49:02.228995 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 22:49:02.231184 18609 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54420:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:49:02.232931 18609 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:49:02.239941 19114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:02.235594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33930 (local address 127.16.121.129:41735)
0928 22:49:02.235778 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:02.235784 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:02.235795 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:49:02.235845 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:02.235847 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:02.235863 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:02.235957 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:49:02.236221 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.237526 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.238691 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.239337 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.239445 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:02.239512 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:02.239779 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 22:49:02.239819 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":68,"threads_started":1}
I20260928 22:49:02.239943 19112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:02.235291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.131:41721 (local address 127.0.0.1:53014)
0928 22:49:02.235509 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:02.235555 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:02.235633 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:02.236248 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:02.236249 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:02.236251 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:49:02.236299 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:02.236300 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.237229 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.237230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:02.238639 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:02.238642 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.239610 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.239611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:02.239672 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:02.239675 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:49:02.239770 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 22:49:02.239813 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":144,"threads_started":1}
I20260928 22:49:02.239954 19115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:02.235584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53014 (local address 127.16.121.131:41721)
0928 22:49:02.236083 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:02.236088 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:02.236097 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:49:02.236134 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:02.236136 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:02.236150 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:02.236385 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:49:02.236448 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.237139 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.238747 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.239523 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.239605 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:02.239661 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:02.239788 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 22:49:02.239849 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":80,"threads_started":1}
I20260928 22:49:02.240105 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:02.235301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:41735 (local address 127.0.0.1:33930)
0928 22:49:02.235728 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:02.235735 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:02.235767 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:02.236000 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:02.236004 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:02.236009 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:49:02.236110 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:02.236112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.237598 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.237599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:02.238604 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:02.238607 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.239446 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.239447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:02.239531 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:02.239551 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 22:49:02.239649 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 22:49:02.240047 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":24,"thread_start_us":83,"threads_started":1}
I20260928 22:49:02.240294 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:02.235395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35994 (local address 127.16.121.130:38211)
0928 22:49:02.235778 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:02.235784 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:02.235795 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:49:02.235859 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:02.235861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:02.235875 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:02.235957 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:49:02.236220 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.237421 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.238747 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.239541 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.239605 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:02.239661 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:02.239991 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 22:49:02.240167 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":84,"threads_started":1}
I20260928 22:49:02.239949 19110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:02.235291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.130:38211 (local address 127.0.0.1:35994)
0928 22:49:02.235558 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:02.235567 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:02.235627 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:02.236000 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:02.236004 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:02.236007 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:49:02.236110 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:02.236113 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.237515 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.237516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:02.238654 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:02.238656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.239698 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.239700 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:02.239774 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:02.239778 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:49:02.239829 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 22:49:02.239871 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":125,"threads_started":1}
I20260928 22:49:02.241068 19027 tablet_service.cc:1467] Processing CreateTablet for tablet f8fd70d493a6438993f49052a74a6be4 (DEFAULT_TABLE table=test-table [id=0f972f4ccc2d4459b584bcf38793fef4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:49:02.241108 18887 tablet_service.cc:1467] Processing CreateTablet for tablet f8fd70d493a6438993f49052a74a6be4 (DEFAULT_TABLE table=test-table [id=0f972f4ccc2d4459b584bcf38793fef4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:49:02.241108 18747 tablet_service.cc:1467] Processing CreateTablet for tablet f8fd70d493a6438993f49052a74a6be4 (DEFAULT_TABLE table=test-table [id=0f972f4ccc2d4459b584bcf38793fef4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:49:02.241441 19027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8fd70d493a6438993f49052a74a6be4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:02.241449 18747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8fd70d493a6438993f49052a74a6be4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:02.241449 18887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8fd70d493a6438993f49052a74a6be4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:02.244240 19116 tablet_bootstrap.cc:492] T f8fd70d493a6438993f49052a74a6be4 P c9a7dce56d024b8998f3a23395492f1c: Bootstrap starting.
I20260928 22:49:02.245124 19116 tablet_bootstrap.cc:654] T f8fd70d493a6438993f49052a74a6be4 P c9a7dce56d024b8998f3a23395492f1c: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:02.245541 19116 log.cc:824] T f8fd70d493a6438993f49052a74a6be4 P c9a7dce56d024b8998f3a23395492f1c: Log is configured to *not* fsync() on all Append() calls
I20260928 22:49:02.245694 19118 tablet_bootstrap.cc:492] T f8fd70d493a6438993f49052a74a6be4 P 5e5a0e809d9842da90487a529361bc7d: Bootstrap starting.
I20260928 22:49:02.246361 19118 tablet_bootstrap.cc:654] T f8fd70d493a6438993f49052a74a6be4 P 5e5a0e809d9842da90487a529361bc7d: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:02.246510 19116 tablet_bootstrap.cc:492] T f8fd70d493a6438993f49052a74a6be4 P c9a7dce56d024b8998f3a23395492f1c: No bootstrap required, opened a new log
I20260928 22:49:02.246598 19116 ts_tablet_manager.cc:1397] T f8fd70d493a6438993f49052a74a6be4 P c9a7dce56d024b8998f3a23395492f1c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:49:02.246721 19118 log.cc:824] T f8fd70d493a6438993f49052a74a6be4 P 5e5a0e809d9842da90487a529361bc7d: Log is configured to *not* fsync() on all Append() calls
I20260928 22:49:02.247169 19117 tablet_bootstrap.cc:492] T f8fd70d493a6438993f49052a74a6be4 P 9d7ed45c903f4384989599e40b950c9f: Bootstrap starting.
I20260928 22:49:02.247740 19117 tablet_bootstrap.cc:654] T f8fd70d493a6438993f49052a74a6be4 P 9d7ed45c903f4384989599e40b950c9f: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:02.248044 19117 log.cc:824] T f8fd70d493a6438993f49052a74a6be4 P 9d7ed45c903f4384989599e40b950c9f: Log is configured to *not* fsync() on all Append() calls
I20260928 22:49:02.248469 19116 raft_consensus.cc:348] T f8fd70d493a6438993f49052a74a6be4 P c9a7dce56d024b8998f3a23395492f1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } } peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } }
I20260928 22:49:02.248642 19118 tablet_bootstrap.cc:492] T f8fd70d493a6438993f49052a74a6be4 P 5e5a0e809d9842da90487a529361bc7d: No bootstrap required, opened a new log
I20260928 22:49:02.248648 19116 raft_consensus.cc:374] T f8fd70d493a6438993f49052a74a6be4 P c9a7dce56d024b8998f3a23395492f1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:02.248692 19116 raft_consensus.cc:729] T f8fd70d493a6438993f49052a74a6be4 P c9a7dce56d024b8998f3a23395492f1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9a7dce56d024b8998f3a23395492f1c, State: Initialized, Role: FOLLOWER
I20260928 22:49:02.248704 19118 ts_tablet_manager.cc:1397] T f8fd70d493a6438993f49052a74a6be4 P 5e5a0e809d9842da90487a529361bc7d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 22:49:02.248837 19116 consensus_queue.cc:260] T f8fd70d493a6438993f49052a74a6be4 P c9a7dce56d024b8998f3a23395492f1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } } peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } }
I20260928 22:49:02.249135 19116 ts_tablet_manager.cc:1428] T f8fd70d493a6438993f49052a74a6be4 P c9a7dce56d024b8998f3a23395492f1c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 22:49:02.249217 19093 heartbeater.cc:503] Master 127.16.121.190:40849 was elected leader, sending a full tablet report...
I20260928 22:49:02.250823 19118 raft_consensus.cc:348] T f8fd70d493a6438993f49052a74a6be4 P 5e5a0e809d9842da90487a529361bc7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } } peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } }
I20260928 22:49:02.251010 19118 raft_consensus.cc:374] T f8fd70d493a6438993f49052a74a6be4 P 5e5a0e809d9842da90487a529361bc7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:02.251096 19118 raft_consensus.cc:729] T f8fd70d493a6438993f49052a74a6be4 P 5e5a0e809d9842da90487a529361bc7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e5a0e809d9842da90487a529361bc7d, State: Initialized, Role: FOLLOWER
I20260928 22:49:02.251232 19118 consensus_queue.cc:260] T f8fd70d493a6438993f49052a74a6be4 P 5e5a0e809d9842da90487a529361bc7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } } peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } }
I20260928 22:49:02.251525 19118 ts_tablet_manager.cc:1428] T f8fd70d493a6438993f49052a74a6be4 P 5e5a0e809d9842da90487a529361bc7d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:49:02.251561 18813 heartbeater.cc:503] Master 127.16.121.190:40849 was elected leader, sending a full tablet report...
I20260928 22:49:02.251760 19117 tablet_bootstrap.cc:492] T f8fd70d493a6438993f49052a74a6be4 P 9d7ed45c903f4384989599e40b950c9f: No bootstrap required, opened a new log
I20260928 22:49:02.251878 19117 ts_tablet_manager.cc:1397] T f8fd70d493a6438993f49052a74a6be4 P 9d7ed45c903f4384989599e40b950c9f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260928 22:49:02.253593 19117 raft_consensus.cc:348] T f8fd70d493a6438993f49052a74a6be4 P 9d7ed45c903f4384989599e40b950c9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } } peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } }
I20260928 22:49:02.253753 19117 raft_consensus.cc:374] T f8fd70d493a6438993f49052a74a6be4 P 9d7ed45c903f4384989599e40b950c9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:02.253800 19117 raft_consensus.cc:729] T f8fd70d493a6438993f49052a74a6be4 P 9d7ed45c903f4384989599e40b950c9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7ed45c903f4384989599e40b950c9f, State: Initialized, Role: FOLLOWER
I20260928 22:49:02.253925 19117 consensus_queue.cc:260] T f8fd70d493a6438993f49052a74a6be4 P 9d7ed45c903f4384989599e40b950c9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } } peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } }
I20260928 22:49:02.254181 19117 ts_tablet_manager.cc:1428] T f8fd70d493a6438993f49052a74a6be4 P 9d7ed45c903f4384989599e40b950c9f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:49:02.254256 18953 heartbeater.cc:503] Master 127.16.121.190:40849 was elected leader, sending a full tablet report...
W20260928 22:49:02.261708 18954 tablet.cc:2369] T f8fd70d493a6438993f49052a74a6be4 P 9d7ed45c903f4384989599e40b950c9f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:49:02.265255 19122 raft_consensus.cc:482] T f8fd70d493a6438993f49052a74a6be4 P c9a7dce56d024b8998f3a23395492f1c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:49:02.265465 19122 raft_consensus.cc:504] T f8fd70d493a6438993f49052a74a6be4 P c9a7dce56d024b8998f3a23395492f1c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } } peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } }
I20260928 22:49:02.265868 19122 leader_election.cc:288] T f8fd70d493a6438993f49052a74a6be4 P c9a7dce56d024b8998f3a23395492f1c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d7ed45c903f4384989599e40b950c9f (127.16.121.130:38211), 5e5a0e809d9842da90487a529361bc7d (127.16.121.129:41735)
I20260928 22:49:02.269353 19125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:02.266102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:41735 (local address 127.16.121.131:45927)
0928 22:49:02.266300 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:02.266311 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:02.266403 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:02.266610 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:02.266611 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:02.266612 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:49:02.266695 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:02.266696 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.267550 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.267551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:02.268435 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:02.268438 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.269066 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.269066 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:02.269137 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:02.269141 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:49:02.269224 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 22:49:02.269263 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":128,"threads_started":1}
I20260928 22:49:02.269340 19114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:02.266207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:45927 (local address 127.16.121.129:41735)
0928 22:49:02.266259 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:02.266261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:02.266390 (+ 129us) server_negotiation.cc:407] Connection header received
0928 22:49:02.266522 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:02.266524 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:02.266534 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:02.266609 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:49:02.266741 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.267470 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.268570 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.268972 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.269065 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:02.269074 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:02.269232 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 22:49:02.269275 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 22:49:02.269457 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:02.266137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:45887 (local address 127.16.121.130:38211)
0928 22:49:02.266195 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:02.266197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:02.266240 (+ 43us) server_negotiation.cc:407] Connection header received
0928 22:49:02.266331 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:02.266332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:02.266341 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:02.266397 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:49:02.266583 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.267389 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.268637 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.269062 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.269155 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:02.269165 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:02.269341 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 22:49:02.269382 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 22:49:02.269456 18986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:02.266073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.130:38211 (local address 127.16.121.131:45887)
0928 22:49:02.266174 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:02.266188 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:02.266255 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:02.266421 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:02.266423 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:02.266424 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:49:02.266507 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:02.266510 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.267497 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.267498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:02.268560 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:02.268563 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.269152 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.269153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:02.269247 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:02.269250 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:49:02.269333 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 22:49:02.269376 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"mutex_wait_us":1}
I20260928 22:49:02.269917 18767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8fd70d493a6438993f49052a74a6be4" candidate_uuid: "c9a7dce56d024b8998f3a23395492f1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e5a0e809d9842da90487a529361bc7d" is_pre_election: true
I20260928 22:49:02.269990 18907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8fd70d493a6438993f49052a74a6be4" candidate_uuid: "c9a7dce56d024b8998f3a23395492f1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7ed45c903f4384989599e40b950c9f" is_pre_election: true
I20260928 22:49:02.270088 18767 raft_consensus.cc:2454] T f8fd70d493a6438993f49052a74a6be4 P 5e5a0e809d9842da90487a529361bc7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9a7dce56d024b8998f3a23395492f1c in term 0.
I20260928 22:49:02.270169 18907 raft_consensus.cc:2454] T f8fd70d493a6438993f49052a74a6be4 P 9d7ed45c903f4384989599e40b950c9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9a7dce56d024b8998f3a23395492f1c in term 0.
I20260928 22:49:02.270344 18980 leader_election.cc:302] T f8fd70d493a6438993f49052a74a6be4 P c9a7dce56d024b8998f3a23395492f1c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e5a0e809d9842da90487a529361bc7d, c9a7dce56d024b8998f3a23395492f1c; no voters:
I20260928 22:49:02.270535 19122 raft_consensus.cc:2781] T f8fd70d493a6438993f49052a74a6be4 P c9a7dce56d024b8998f3a23395492f1c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:49:02.270601 19122 raft_consensus.cc:482] T f8fd70d493a6438993f49052a74a6be4 P c9a7dce56d024b8998f3a23395492f1c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:49:02.270618 19122 raft_consensus.cc:3037] T f8fd70d493a6438993f49052a74a6be4 P c9a7dce56d024b8998f3a23395492f1c [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:02.271409 19122 raft_consensus.cc:504] T f8fd70d493a6438993f49052a74a6be4 P c9a7dce56d024b8998f3a23395492f1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } } peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } }
I20260928 22:49:02.271569 19122 leader_election.cc:288] T f8fd70d493a6438993f49052a74a6be4 P c9a7dce56d024b8998f3a23395492f1c [CANDIDATE]: Term 1 election: Requested vote from peers 9d7ed45c903f4384989599e40b950c9f (127.16.121.130:38211), 5e5a0e809d9842da90487a529361bc7d (127.16.121.129:41735)
I20260928 22:49:02.271727 18907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8fd70d493a6438993f49052a74a6be4" candidate_uuid: "c9a7dce56d024b8998f3a23395492f1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7ed45c903f4384989599e40b950c9f"
I20260928 22:49:02.271766 18767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8fd70d493a6438993f49052a74a6be4" candidate_uuid: "c9a7dce56d024b8998f3a23395492f1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e5a0e809d9842da90487a529361bc7d"
I20260928 22:49:02.271842 18907 raft_consensus.cc:3037] T f8fd70d493a6438993f49052a74a6be4 P 9d7ed45c903f4384989599e40b950c9f [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:02.271842 18767 raft_consensus.cc:3037] T f8fd70d493a6438993f49052a74a6be4 P 5e5a0e809d9842da90487a529361bc7d [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:02.272586 18907 raft_consensus.cc:2454] T f8fd70d493a6438993f49052a74a6be4 P 9d7ed45c903f4384989599e40b950c9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9a7dce56d024b8998f3a23395492f1c in term 1.
I20260928 22:49:02.272650 18767 raft_consensus.cc:2454] T f8fd70d493a6438993f49052a74a6be4 P 5e5a0e809d9842da90487a529361bc7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9a7dce56d024b8998f3a23395492f1c in term 1.
I20260928 22:49:02.273136 18979 leader_election.cc:302] T f8fd70d493a6438993f49052a74a6be4 P c9a7dce56d024b8998f3a23395492f1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d7ed45c903f4384989599e40b950c9f, c9a7dce56d024b8998f3a23395492f1c; no voters:
I20260928 22:49:02.273304 19122 raft_consensus.cc:2781] T f8fd70d493a6438993f49052a74a6be4 P c9a7dce56d024b8998f3a23395492f1c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:49:02.273510 19122 raft_consensus.cc:686] T f8fd70d493a6438993f49052a74a6be4 P c9a7dce56d024b8998f3a23395492f1c [term 1 LEADER]: Becoming Leader. State: Replica: c9a7dce56d024b8998f3a23395492f1c, State: Running, Role: LEADER
I20260928 22:49:02.273634 19122 consensus_queue.cc:237] T f8fd70d493a6438993f49052a74a6be4 P c9a7dce56d024b8998f3a23395492f1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } } peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } }
I20260928 22:49:02.274526 18609 catalog_manager.cc:5495] T f8fd70d493a6438993f49052a74a6be4 P c9a7dce56d024b8998f3a23395492f1c reported cstate change: term changed from 0 to 1, leader changed from <none> to c9a7dce56d024b8998f3a23395492f1c (127.16.121.131). New cstate: current_term: 1 leader_uuid: "c9a7dce56d024b8998f3a23395492f1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } health_report { overall_health: UNKNOWN } } }
Sep 28 22:49:02 dist-test-slave-56vw krb5kdc[18565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635741, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:49:02.299358 19133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:02.291917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:40849 (local address 127.0.0.1:54432)
0928 22:49:02.292082 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:02.292095 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:02.292161 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:02.292346 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:02.292347 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:02.292682 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:02.292734 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:02.292737 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.293446 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.293447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:02.294090 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:02.294092 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.294505 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.294506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:02.294608 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:02.295265 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:02.295269 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:02.296707 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:02.298266 (+ 1559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:02.298273 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:02.298275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:02.298538 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:02.298734 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:02.298735 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:02.298735 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:02.298793 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:02.299004 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:02.299010 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:02.299110 (+ 100us) client_negotiation.cc:769] Sending connection context
0928 22:49:02.299183 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 22:49:02.299284 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":107,"threads_started":1}
I20260928 22:49:02.299453 18667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:02.291978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54432 (local address 127.16.121.190:40849)
0928 22:49:02.292049 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:02.292051 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:02.292159 (+ 108us) server_negotiation.cc:407] Connection header received
0928 22:49:02.292205 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:02.292206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:02.292215 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:02.292273 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:02.292851 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.293345 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.294188 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.294422 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.294492 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:02.296791 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:02.296792 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:02.296793 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:02.296799 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:02.298170 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:02.298618 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:02.298618 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:02.298619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:02.298671 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:02.298846 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:02.298847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:02.298847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:02.298947 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:02.299003 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:02.299217 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 22:49:02.299339 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":38}
I20260928 22:49:02.301659 18609 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:54432:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 22:49:02.301975 18609 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:49:02.303601 18747 tablet_service.cc:1467] Processing CreateTablet for tablet 2c5cd423d9ea401eb8968d25b99495c7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e92fde8c0e6b4ec9a10b46396cbe8b30]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 22:49:02.303791 18747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c5cd423d9ea401eb8968d25b99495c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:02.303732 19027 tablet_service.cc:1467] Processing CreateTablet for tablet 2c5cd423d9ea401eb8968d25b99495c7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e92fde8c0e6b4ec9a10b46396cbe8b30]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 22:49:02.303916 19027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c5cd423d9ea401eb8968d25b99495c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:02.305194 19118 tablet_bootstrap.cc:492] T 2c5cd423d9ea401eb8968d25b99495c7 P 5e5a0e809d9842da90487a529361bc7d: Bootstrap starting.
I20260928 22:49:02.305233 19116 tablet_bootstrap.cc:492] T 2c5cd423d9ea401eb8968d25b99495c7 P c9a7dce56d024b8998f3a23395492f1c: Bootstrap starting.
I20260928 22:49:02.305337 18887 tablet_service.cc:1467] Processing CreateTablet for tablet 2c5cd423d9ea401eb8968d25b99495c7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e92fde8c0e6b4ec9a10b46396cbe8b30]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 22:49:02.305469 18887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c5cd423d9ea401eb8968d25b99495c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:02.305702 19116 tablet_bootstrap.cc:654] T 2c5cd423d9ea401eb8968d25b99495c7 P c9a7dce56d024b8998f3a23395492f1c: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:02.305958 19118 tablet_bootstrap.cc:654] T 2c5cd423d9ea401eb8968d25b99495c7 P 5e5a0e809d9842da90487a529361bc7d: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:02.306419 19116 tablet_bootstrap.cc:492] T 2c5cd423d9ea401eb8968d25b99495c7 P c9a7dce56d024b8998f3a23395492f1c: No bootstrap required, opened a new log
I20260928 22:49:02.306488 19116 ts_tablet_manager.cc:1397] T 2c5cd423d9ea401eb8968d25b99495c7 P c9a7dce56d024b8998f3a23395492f1c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:02.306559 19118 tablet_bootstrap.cc:492] T 2c5cd423d9ea401eb8968d25b99495c7 P 5e5a0e809d9842da90487a529361bc7d: No bootstrap required, opened a new log
I20260928 22:49:02.306608 19118 ts_tablet_manager.cc:1397] T 2c5cd423d9ea401eb8968d25b99495c7 P 5e5a0e809d9842da90487a529361bc7d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:02.306639 19116 raft_consensus.cc:348] T 2c5cd423d9ea401eb8968d25b99495c7 P c9a7dce56d024b8998f3a23395492f1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } }
I20260928 22:49:02.306713 19116 raft_consensus.cc:374] T 2c5cd423d9ea401eb8968d25b99495c7 P c9a7dce56d024b8998f3a23395492f1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:02.306740 19117 tablet_bootstrap.cc:492] T 2c5cd423d9ea401eb8968d25b99495c7 P 9d7ed45c903f4384989599e40b950c9f: Bootstrap starting.
I20260928 22:49:02.306756 19116 raft_consensus.cc:729] T 2c5cd423d9ea401eb8968d25b99495c7 P c9a7dce56d024b8998f3a23395492f1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9a7dce56d024b8998f3a23395492f1c, State: Initialized, Role: FOLLOWER
I20260928 22:49:02.306766 19118 raft_consensus.cc:348] T 2c5cd423d9ea401eb8968d25b99495c7 P 5e5a0e809d9842da90487a529361bc7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } }
I20260928 22:49:02.306805 19116 consensus_queue.cc:260] T 2c5cd423d9ea401eb8968d25b99495c7 P c9a7dce56d024b8998f3a23395492f1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } }
I20260928 22:49:02.306838 19118 raft_consensus.cc:374] T 2c5cd423d9ea401eb8968d25b99495c7 P 5e5a0e809d9842da90487a529361bc7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:02.306861 19118 raft_consensus.cc:729] T 2c5cd423d9ea401eb8968d25b99495c7 P 5e5a0e809d9842da90487a529361bc7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e5a0e809d9842da90487a529361bc7d, State: Initialized, Role: FOLLOWER
I20260928 22:49:02.306921 19116 ts_tablet_manager.cc:1428] T 2c5cd423d9ea401eb8968d25b99495c7 P c9a7dce56d024b8998f3a23395492f1c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 22:49:02.306921 19118 consensus_queue.cc:260] T 2c5cd423d9ea401eb8968d25b99495c7 P 5e5a0e809d9842da90487a529361bc7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } }
I20260928 22:49:02.307034 19118 ts_tablet_manager.cc:1428] T 2c5cd423d9ea401eb8968d25b99495c7 P 5e5a0e809d9842da90487a529361bc7d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 22:49:02.307031 19122 tablet_replica.cc:439] T 2c5cd423d9ea401eb8968d25b99495c7 P c9a7dce56d024b8998f3a23395492f1c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } } }
I20260928 22:49:02.307137 19122 tablet_replica.cc:442] T 2c5cd423d9ea401eb8968d25b99495c7 P c9a7dce56d024b8998f3a23395492f1c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:49:02.307216 19123 tablet_replica.cc:439] T 2c5cd423d9ea401eb8968d25b99495c7 P 5e5a0e809d9842da90487a529361bc7d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } } }
I20260928 22:49:02.307287 19123 tablet_replica.cc:442] T 2c5cd423d9ea401eb8968d25b99495c7 P 5e5a0e809d9842da90487a529361bc7d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:49:02.307480 19117 tablet_bootstrap.cc:654] T 2c5cd423d9ea401eb8968d25b99495c7 P 9d7ed45c903f4384989599e40b950c9f: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:02.308162 19117 tablet_bootstrap.cc:492] T 2c5cd423d9ea401eb8968d25b99495c7 P 9d7ed45c903f4384989599e40b950c9f: No bootstrap required, opened a new log
I20260928 22:49:02.308203 19117 ts_tablet_manager.cc:1397] T 2c5cd423d9ea401eb8968d25b99495c7 P 9d7ed45c903f4384989599e40b950c9f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:02.308357 19117 raft_consensus.cc:348] T 2c5cd423d9ea401eb8968d25b99495c7 P 9d7ed45c903f4384989599e40b950c9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } }
I20260928 22:49:02.308490 19117 raft_consensus.cc:374] T 2c5cd423d9ea401eb8968d25b99495c7 P 9d7ed45c903f4384989599e40b950c9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:02.308506 19117 raft_consensus.cc:729] T 2c5cd423d9ea401eb8968d25b99495c7 P 9d7ed45c903f4384989599e40b950c9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7ed45c903f4384989599e40b950c9f, State: Initialized, Role: FOLLOWER
I20260928 22:49:02.308578 19117 consensus_queue.cc:260] T 2c5cd423d9ea401eb8968d25b99495c7 P 9d7ed45c903f4384989599e40b950c9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } }
I20260928 22:49:02.308668 19117 ts_tablet_manager.cc:1428] T 2c5cd423d9ea401eb8968d25b99495c7 P 9d7ed45c903f4384989599e40b950c9f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 22:49:02.308825 19124 tablet_replica.cc:439] T 2c5cd423d9ea401eb8968d25b99495c7 P 9d7ed45c903f4384989599e40b950c9f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } } }
I20260928 22:49:02.308902 19124 tablet_replica.cc:442] T 2c5cd423d9ea401eb8968d25b99495c7 P 9d7ed45c903f4384989599e40b950c9f: This TxnStatusTablet replica's current role is: FOLLOWER
W20260928 22:49:02.336753 18814 tablet.cc:2369] T 2c5cd423d9ea401eb8968d25b99495c7 P 5e5a0e809d9842da90487a529361bc7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:49:02.429876 19123 raft_consensus.cc:482] T 2c5cd423d9ea401eb8968d25b99495c7 P 5e5a0e809d9842da90487a529361bc7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:49:02.429986 19123 raft_consensus.cc:504] T 2c5cd423d9ea401eb8968d25b99495c7 P 5e5a0e809d9842da90487a529361bc7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } }
I20260928 22:49:02.430310 19123 leader_election.cc:288] T 2c5cd423d9ea401eb8968d25b99495c7 P 5e5a0e809d9842da90487a529361bc7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d7ed45c903f4384989599e40b950c9f (127.16.121.130:38211), c9a7dce56d024b8998f3a23395492f1c (127.16.121.131:41721)
I20260928 22:49:02.434852 19115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:02.430755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:40123 (local address 127.16.121.131:41721)
0928 22:49:02.430807 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:02.430809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:02.430819 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:49:02.430921 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:02.430921 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:02.430930 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:02.431010 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:49:02.431278 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.432067 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.433718 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.434294 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.434479 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:02.434486 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:02.434726 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 22:49:02.434774 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 22:49:02.434865 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:02.430679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:60107 (local address 127.16.121.130:38211)
0928 22:49:02.430759 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:02.430761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:02.430810 (+ 49us) server_negotiation.cc:407] Connection header received
0928 22:49:02.430918 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:02.430919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:02.430930 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:02.431010 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:49:02.431277 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.432058 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.433718 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.434294 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.434378 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:02.434388 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:02.434728 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 22:49:02.434776 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 22:49:02.434861 19138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:02.430586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.131:41721 (local address 127.16.121.129:40123)
0928 22:49:02.430731 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:02.430744 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:02.430840 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:02.431074 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:02.431076 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:02.431081 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:49:02.431183 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:02.431185 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.432166 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.432167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:02.433597 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:02.433602 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.434546 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.434546 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:02.434610 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:02.434613 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:49:02.434672 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 22:49:02.434743 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":150,"thread_start_us":73,"threads_started":1}
I20260928 22:49:02.434870 19137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:02.430586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.130:38211 (local address 127.16.121.129:60107)
0928 22:49:02.430743 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:02.430753 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:02.430822 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:02.431074 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:02.431076 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:02.431079 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:49:02.431165 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:02.431167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.432166 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.432167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:02.433598 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:02.433602 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.434489 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.434490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:02.434601 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:02.434606 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:49:02.434672 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 22:49:02.434743 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":87,"thread_start_us":56,"threads_started":1}
I20260928 22:49:02.435257 19047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5cd423d9ea401eb8968d25b99495c7" candidate_uuid: "5e5a0e809d9842da90487a529361bc7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a7dce56d024b8998f3a23395492f1c" is_pre_election: true
I20260928 22:49:02.435266 18907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5cd423d9ea401eb8968d25b99495c7" candidate_uuid: "5e5a0e809d9842da90487a529361bc7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7ed45c903f4384989599e40b950c9f" is_pre_election: true
I20260928 22:49:02.435369 18907 raft_consensus.cc:2454] T 2c5cd423d9ea401eb8968d25b99495c7 P 9d7ed45c903f4384989599e40b950c9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e5a0e809d9842da90487a529361bc7d in term 0.
I20260928 22:49:02.435393 19047 raft_consensus.cc:2454] T 2c5cd423d9ea401eb8968d25b99495c7 P c9a7dce56d024b8998f3a23395492f1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e5a0e809d9842da90487a529361bc7d in term 0.
I20260928 22:49:02.435561 18699 leader_election.cc:302] T 2c5cd423d9ea401eb8968d25b99495c7 P 5e5a0e809d9842da90487a529361bc7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e5a0e809d9842da90487a529361bc7d, 9d7ed45c903f4384989599e40b950c9f; no voters:
I20260928 22:49:02.435765 19123 raft_consensus.cc:2781] T 2c5cd423d9ea401eb8968d25b99495c7 P 5e5a0e809d9842da90487a529361bc7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:49:02.435822 19123 raft_consensus.cc:482] T 2c5cd423d9ea401eb8968d25b99495c7 P 5e5a0e809d9842da90487a529361bc7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:49:02.435856 19123 raft_consensus.cc:3037] T 2c5cd423d9ea401eb8968d25b99495c7 P 5e5a0e809d9842da90487a529361bc7d [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:02.436595 19123 raft_consensus.cc:504] T 2c5cd423d9ea401eb8968d25b99495c7 P 5e5a0e809d9842da90487a529361bc7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } }
I20260928 22:49:02.436761 19123 leader_election.cc:288] T 2c5cd423d9ea401eb8968d25b99495c7 P 5e5a0e809d9842da90487a529361bc7d [CANDIDATE]: Term 1 election: Requested vote from peers 9d7ed45c903f4384989599e40b950c9f (127.16.121.130:38211), c9a7dce56d024b8998f3a23395492f1c (127.16.121.131:41721)
I20260928 22:49:02.436952 19047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5cd423d9ea401eb8968d25b99495c7" candidate_uuid: "5e5a0e809d9842da90487a529361bc7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a7dce56d024b8998f3a23395492f1c"
I20260928 22:49:02.437037 19047 raft_consensus.cc:3037] T 2c5cd423d9ea401eb8968d25b99495c7 P c9a7dce56d024b8998f3a23395492f1c [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:02.436997 18907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5cd423d9ea401eb8968d25b99495c7" candidate_uuid: "5e5a0e809d9842da90487a529361bc7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7ed45c903f4384989599e40b950c9f"
I20260928 22:49:02.437089 18907 raft_consensus.cc:3037] T 2c5cd423d9ea401eb8968d25b99495c7 P 9d7ed45c903f4384989599e40b950c9f [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:02.437707 18907 raft_consensus.cc:2454] T 2c5cd423d9ea401eb8968d25b99495c7 P 9d7ed45c903f4384989599e40b950c9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e5a0e809d9842da90487a529361bc7d in term 1.
I20260928 22:49:02.437906 18699 leader_election.cc:302] T 2c5cd423d9ea401eb8968d25b99495c7 P 5e5a0e809d9842da90487a529361bc7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e5a0e809d9842da90487a529361bc7d, 9d7ed45c903f4384989599e40b950c9f; no voters:
I20260928 22:49:02.437911 19047 raft_consensus.cc:2454] T 2c5cd423d9ea401eb8968d25b99495c7 P c9a7dce56d024b8998f3a23395492f1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e5a0e809d9842da90487a529361bc7d in term 1.
I20260928 22:49:02.438006 19123 raft_consensus.cc:2781] T 2c5cd423d9ea401eb8968d25b99495c7 P 5e5a0e809d9842da90487a529361bc7d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:49:02.438119 19123 raft_consensus.cc:686] T 2c5cd423d9ea401eb8968d25b99495c7 P 5e5a0e809d9842da90487a529361bc7d [term 1 LEADER]: Becoming Leader. State: Replica: 5e5a0e809d9842da90487a529361bc7d, State: Running, Role: LEADER
I20260928 22:49:02.438241 19123 consensus_queue.cc:237] T 2c5cd423d9ea401eb8968d25b99495c7 P 5e5a0e809d9842da90487a529361bc7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } }
I20260928 22:49:02.438716 19139 tablet_replica.cc:439] T 2c5cd423d9ea401eb8968d25b99495c7 P 5e5a0e809d9842da90487a529361bc7d: TxnStatusTablet state changed. Reason: New leader 5e5a0e809d9842da90487a529361bc7d. Latest consensus state: current_term: 1 leader_uuid: "5e5a0e809d9842da90487a529361bc7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } } }
I20260928 22:49:02.438810 19139 tablet_replica.cc:442] T 2c5cd423d9ea401eb8968d25b99495c7 P 5e5a0e809d9842da90487a529361bc7d: This TxnStatusTablet replica's current role is: LEADER
I20260928 22:49:02.438994 19141 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 22:49:02.439149 19141 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 22:49:02.439186 18610 catalog_manager.cc:5495] T 2c5cd423d9ea401eb8968d25b99495c7 P 5e5a0e809d9842da90487a529361bc7d reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e5a0e809d9842da90487a529361bc7d (127.16.121.129). New cstate: current_term: 1 leader_uuid: "5e5a0e809d9842da90487a529361bc7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } health_report { overall_health: UNKNOWN } } }
W20260928 22:49:02.447263 19094 tablet.cc:2369] T 2c5cd423d9ea401eb8968d25b99495c7 P c9a7dce56d024b8998f3a23395492f1c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:49:02.456619 19114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:02.453326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33932 (local address 127.16.121.129:41735)
0928 22:49:02.453396 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:02.453399 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:02.453461 (+ 62us) server_negotiation.cc:407] Connection header received
0928 22:49:02.453570 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:02.453571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:02.453588 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:02.453649 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:49:02.453959 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.454468 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.455706 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.455916 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.455977 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:02.456220 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:49:02.456356 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:49:02.456456 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:02.456525 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 22:49:02.456549 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 22:49:02.456660 19133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:02.453262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:41735 (local address 127.0.0.1:33932)
0928 22:49:02.453400 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:02.453414 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:02.453486 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:02.453736 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:02.453737 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:02.453739 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 22:49:02.453872 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:02.453875 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.454558 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.454559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:02.455606 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:02.455609 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.455999 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.456000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:02.456072 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:02.456091 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 22:49:02.456450 (+ 359us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 22:49:02.456460 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 22:49:02.456519 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 22:49:02.456571 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260928 22:49:02.458722 18907 raft_consensus.cc:1261] T 2c5cd423d9ea401eb8968d25b99495c7 P 9d7ed45c903f4384989599e40b950c9f [term 1 FOLLOWER]: Refusing update from remote peer 5e5a0e809d9842da90487a529361bc7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:49:02.458814 19047 raft_consensus.cc:1261] T 2c5cd423d9ea401eb8968d25b99495c7 P c9a7dce56d024b8998f3a23395492f1c [term 1 FOLLOWER]: Refusing update from remote peer 5e5a0e809d9842da90487a529361bc7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:49:02.459195 19123 consensus_queue.cc:1035] T 2c5cd423d9ea401eb8968d25b99495c7 P 5e5a0e809d9842da90487a529361bc7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:49:02.459378 19139 consensus_queue.cc:1035] T 2c5cd423d9ea401eb8968d25b99495c7 P 5e5a0e809d9842da90487a529361bc7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:49:02.460865 19122 tablet_replica.cc:439] T 2c5cd423d9ea401eb8968d25b99495c7 P c9a7dce56d024b8998f3a23395492f1c: TxnStatusTablet state changed. Reason: New leader 5e5a0e809d9842da90487a529361bc7d. Latest consensus state: current_term: 1 leader_uuid: "5e5a0e809d9842da90487a529361bc7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } } }
I20260928 22:49:02.460971 19122 tablet_replica.cc:442] T 2c5cd423d9ea401eb8968d25b99495c7 P c9a7dce56d024b8998f3a23395492f1c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:49:02.461097 19124 tablet_replica.cc:439] T 2c5cd423d9ea401eb8968d25b99495c7 P 9d7ed45c903f4384989599e40b950c9f: TxnStatusTablet state changed. Reason: New leader 5e5a0e809d9842da90487a529361bc7d. Latest consensus state: current_term: 1 leader_uuid: "5e5a0e809d9842da90487a529361bc7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } } }
I20260928 22:49:02.461197 19124 tablet_replica.cc:442] T 2c5cd423d9ea401eb8968d25b99495c7 P 9d7ed45c903f4384989599e40b950c9f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:49:02.462087 19123 tablet_replica.cc:439] T 2c5cd423d9ea401eb8968d25b99495c7 P 5e5a0e809d9842da90487a529361bc7d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5e5a0e809d9842da90487a529361bc7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } } }
I20260928 22:49:02.462194 19123 tablet_replica.cc:442] T 2c5cd423d9ea401eb8968d25b99495c7 P 5e5a0e809d9842da90487a529361bc7d: This TxnStatusTablet replica's current role is: LEADER
I20260928 22:49:02.462272 19123 tablet_replica.cc:439] T 2c5cd423d9ea401eb8968d25b99495c7 P 5e5a0e809d9842da90487a529361bc7d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5e5a0e809d9842da90487a529361bc7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } } }
I20260928 22:49:02.462352 19123 tablet_replica.cc:442] T 2c5cd423d9ea401eb8968d25b99495c7 P 5e5a0e809d9842da90487a529361bc7d: This TxnStatusTablet replica's current role is: LEADER
I20260928 22:49:02.462352 19124 tablet_replica.cc:439] T 2c5cd423d9ea401eb8968d25b99495c7 P 9d7ed45c903f4384989599e40b950c9f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5e5a0e809d9842da90487a529361bc7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } } }
I20260928 22:49:02.462436 19124 tablet_replica.cc:442] T 2c5cd423d9ea401eb8968d25b99495c7 P 9d7ed45c903f4384989599e40b950c9f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:49:02.462800 19122 tablet_replica.cc:439] T 2c5cd423d9ea401eb8968d25b99495c7 P c9a7dce56d024b8998f3a23395492f1c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5e5a0e809d9842da90487a529361bc7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } } }
I20260928 22:49:02.462890 19122 tablet_replica.cc:442] T 2c5cd423d9ea401eb8968d25b99495c7 P c9a7dce56d024b8998f3a23395492f1c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:49:02.463251 19146 mvcc.cc:205] Tried to move back new op lower bound from 7334444001107685376 to 7334444001027510272. Current Snapshot: MvccSnapshot[applied={T|T < 7334444001107685376}]
I20260928 22:49:02.464442 19145 mvcc.cc:205] Tried to move back new op lower bound from 7334444001107685376 to 7334444001027510272. Current Snapshot: MvccSnapshot[applied={T|T < 7334444001107685376 or (T in {7334444001107685376})}]
I20260928 22:49:02.468437 19115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:02.465428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53022 (local address 127.16.121.131:41721)
0928 22:49:02.465466 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:02.465468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:02.465477 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:49:02.465569 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:02.465570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:02.465587 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:02.465677 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:49:02.465838 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.466377 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.467451 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.467716 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.467811 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:02.468016 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:49:02.468202 (+ 186us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:49:02.468295 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:02.468331 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 22:49:02.468352 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 22:49:02.473443 19114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:02.471086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:49311 (local address 127.16.121.129:41735)
0928 22:49:02.471135 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:02.471137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:02.471168 (+ 31us) server_negotiation.cc:407] Connection header received
0928 22:49:02.471192 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:02.471193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:02.471203 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:02.471255 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:49:02.471448 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.471925 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.472706 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.472906 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.472998 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:02.473185 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:49:02.473224 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:49:02.473286 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:02.473339 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 22:49:02.473359 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 22:49:02.473445 19099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:02.470960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:41735 (local address 127.16.121.131:49311)
0928 22:49:02.471067 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:02.471079 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:02.471129 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:02.471258 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:02.471258 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:02.471259 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 22:49:02.471320 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:02.471321 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.472025 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.472026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:02.472608 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:02.472610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.473000 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.473001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:02.473066 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:02.473087 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 22:49:02.473286 (+ 199us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 22:49:02.473288 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 22:49:02.473336 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 22:49:02.473382 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 22:49:02.476255 18767 raft_consensus.cc:1261] T f8fd70d493a6438993f49052a74a6be4 P 5e5a0e809d9842da90487a529361bc7d [term 1 FOLLOWER]: Refusing update from remote peer c9a7dce56d024b8998f3a23395492f1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:49:02.476460 18907 raft_consensus.cc:1261] T f8fd70d493a6438993f49052a74a6be4 P 9d7ed45c903f4384989599e40b950c9f [term 1 FOLLOWER]: Refusing update from remote peer c9a7dce56d024b8998f3a23395492f1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:49:02.476589 19126 consensus_queue.cc:1035] T f8fd70d493a6438993f49052a74a6be4 P c9a7dce56d024b8998f3a23395492f1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:49:02.476902 19126 consensus_queue.cc:1035] T f8fd70d493a6438993f49052a74a6be4 P c9a7dce56d024b8998f3a23395492f1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:49:02.478040 19144 mvcc.cc:205] Tried to move back new op lower bound from 7334444001180635136 to 7334444000353693696. Current Snapshot: MvccSnapshot[applied={T|T < 7334444001180635136}]
I20260928 22:49:02.486495 19115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:02.483588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:48495 (local address 127.16.121.131:41721)
0928 22:49:02.483635 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:02.483637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:02.483809 (+ 172us) server_negotiation.cc:407] Connection header received
0928 22:49:02.483873 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:02.483874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:02.483884 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:02.483939 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:49:02.484091 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.484570 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.485737 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.486004 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.486087 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:02.486255 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:49:02.486289 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:49:02.486337 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:02.486387 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 22:49:02.486414 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:49:02.486655 19160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:02.483557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.131:41721 (local address 127.16.121.129:48495)
0928 22:49:02.483733 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:02.483743 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:02.483820 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:02.483943 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:02.483944 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:02.483945 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 22:49:02.484015 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:02.484017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.484653 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.484654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:02.485655 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:02.485658 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.486088 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.486089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:02.486153 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:02.486170 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 22:49:02.486338 (+ 168us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 22:49:02.486340 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 22:49:02.486385 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 22:49:02.486427 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":97,"threads_started":1}
I20260928 22:49:02.495569 18610 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:49:02.495707 18610 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:49:02.496609 18610 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 622784bfe8d5446abd97cdd40c6d8115: Sending DeleteTablet for 3 replicas of tablet f8fd70d493a6438993f49052a74a6be4
I20260928 22:49:02.496985 19027 tablet_service.cc:1514] Processing DeleteTablet for tablet f8fd70d493a6438993f49052a74a6be4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:49:02 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:53014
I20260928 22:49:02.497210 18887 tablet_service.cc:1514] Processing DeleteTablet for tablet f8fd70d493a6438993f49052a74a6be4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:49:02 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:35994
I20260928 22:49:02.497468 18747 tablet_service.cc:1514] Processing DeleteTablet for tablet f8fd70d493a6438993f49052a74a6be4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:49:02 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:33930
I20260928 22:49:02.497956 18610 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54420:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:49:02.498099 18610 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:49:02.499603 19027 tablet_service.cc:1467] Processing CreateTablet for tablet f65aaa316d8f41709def1ed95149d69a (DEFAULT_TABLE table=test-table [id=f1e4ef48f49d47309f11c7875f6f7b46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:49:02.499723 18887 tablet_service.cc:1467] Processing CreateTablet for tablet f65aaa316d8f41709def1ed95149d69a (DEFAULT_TABLE table=test-table [id=f1e4ef48f49d47309f11c7875f6f7b46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:49:02.499754 19027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f65aaa316d8f41709def1ed95149d69a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:02.499871 18887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f65aaa316d8f41709def1ed95149d69a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:02.501271 19117 tablet_bootstrap.cc:492] T f65aaa316d8f41709def1ed95149d69a P 9d7ed45c903f4384989599e40b950c9f: Bootstrap starting.
I20260928 22:49:02.501271 19116 tablet_bootstrap.cc:492] T f65aaa316d8f41709def1ed95149d69a P c9a7dce56d024b8998f3a23395492f1c: Bootstrap starting.
I20260928 22:49:02.501722 19117 tablet_bootstrap.cc:654] T f65aaa316d8f41709def1ed95149d69a P 9d7ed45c903f4384989599e40b950c9f: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:02.501730 19116 tablet_bootstrap.cc:654] T f65aaa316d8f41709def1ed95149d69a P c9a7dce56d024b8998f3a23395492f1c: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:02.502279 18747 tablet_service.cc:1467] Processing CreateTablet for tablet f65aaa316d8f41709def1ed95149d69a (DEFAULT_TABLE table=test-table [id=f1e4ef48f49d47309f11c7875f6f7b46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:49:02.502429 18747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f65aaa316d8f41709def1ed95149d69a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:02.502774 19116 tablet_bootstrap.cc:492] T f65aaa316d8f41709def1ed95149d69a P c9a7dce56d024b8998f3a23395492f1c: No bootstrap required, opened a new log
I20260928 22:49:02.502833 19116 ts_tablet_manager.cc:1397] T f65aaa316d8f41709def1ed95149d69a P c9a7dce56d024b8998f3a23395492f1c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:49:02.502996 19116 raft_consensus.cc:348] T f65aaa316d8f41709def1ed95149d69a P c9a7dce56d024b8998f3a23395492f1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } }
I20260928 22:49:02.503201 19116 raft_consensus.cc:374] T f65aaa316d8f41709def1ed95149d69a P c9a7dce56d024b8998f3a23395492f1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:02.503240 19116 raft_consensus.cc:729] T f65aaa316d8f41709def1ed95149d69a P c9a7dce56d024b8998f3a23395492f1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9a7dce56d024b8998f3a23395492f1c, State: Initialized, Role: FOLLOWER
I20260928 22:49:02.503290 19116 consensus_queue.cc:260] T f65aaa316d8f41709def1ed95149d69a P c9a7dce56d024b8998f3a23395492f1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } }
I20260928 22:49:02.503407 19116 ts_tablet_manager.cc:1428] T f65aaa316d8f41709def1ed95149d69a P c9a7dce56d024b8998f3a23395492f1c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 22:49:02.503786 19118 tablet_bootstrap.cc:492] T f65aaa316d8f41709def1ed95149d69a P 5e5a0e809d9842da90487a529361bc7d: Bootstrap starting.
I20260928 22:49:02.504233 19118 tablet_bootstrap.cc:654] T f65aaa316d8f41709def1ed95149d69a P 5e5a0e809d9842da90487a529361bc7d: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:02.504850 19117 tablet_bootstrap.cc:492] T f65aaa316d8f41709def1ed95149d69a P 9d7ed45c903f4384989599e40b950c9f: No bootstrap required, opened a new log
I20260928 22:49:02.504902 19117 ts_tablet_manager.cc:1397] T f65aaa316d8f41709def1ed95149d69a P 9d7ed45c903f4384989599e40b950c9f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 22:49:02.504998 19118 tablet_bootstrap.cc:492] T f65aaa316d8f41709def1ed95149d69a P 5e5a0e809d9842da90487a529361bc7d: No bootstrap required, opened a new log
I20260928 22:49:02.505041 19118 ts_tablet_manager.cc:1397] T f65aaa316d8f41709def1ed95149d69a P 5e5a0e809d9842da90487a529361bc7d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 22:49:02.505061 19117 raft_consensus.cc:348] T f65aaa316d8f41709def1ed95149d69a P 9d7ed45c903f4384989599e40b950c9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } }
I20260928 22:49:02.505128 19117 raft_consensus.cc:374] T f65aaa316d8f41709def1ed95149d69a P 9d7ed45c903f4384989599e40b950c9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:02.505148 19117 raft_consensus.cc:729] T f65aaa316d8f41709def1ed95149d69a P 9d7ed45c903f4384989599e40b950c9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7ed45c903f4384989599e40b950c9f, State: Initialized, Role: FOLLOWER
I20260928 22:49:02.505187 19117 consensus_queue.cc:260] T f65aaa316d8f41709def1ed95149d69a P 9d7ed45c903f4384989599e40b950c9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } }
I20260928 22:49:02.505430 19118 raft_consensus.cc:348] T f65aaa316d8f41709def1ed95149d69a P 5e5a0e809d9842da90487a529361bc7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } }
I20260928 22:49:02.505515 19118 raft_consensus.cc:374] T f65aaa316d8f41709def1ed95149d69a P 5e5a0e809d9842da90487a529361bc7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:02.505537 19118 raft_consensus.cc:729] T f65aaa316d8f41709def1ed95149d69a P 5e5a0e809d9842da90487a529361bc7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e5a0e809d9842da90487a529361bc7d, State: Initialized, Role: FOLLOWER
I20260928 22:49:02.505579 19118 consensus_queue.cc:260] T f65aaa316d8f41709def1ed95149d69a P 5e5a0e809d9842da90487a529361bc7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } }
I20260928 22:49:02.505671 19118 ts_tablet_manager.cc:1428] T f65aaa316d8f41709def1ed95149d69a P 5e5a0e809d9842da90487a529361bc7d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 22:49:02.505266 19117 ts_tablet_manager.cc:1428] T f65aaa316d8f41709def1ed95149d69a P 9d7ed45c903f4384989599e40b950c9f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 22:49:02.507889 19163 tablet_replica.cc:330] T f8fd70d493a6438993f49052a74a6be4 P c9a7dce56d024b8998f3a23395492f1c: stopping tablet replica
I20260928 22:49:02.508024 19163 raft_consensus.cc:2229] T f8fd70d493a6438993f49052a74a6be4 P c9a7dce56d024b8998f3a23395492f1c [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:49:02.508462 19164 tablet_replica.cc:330] T f8fd70d493a6438993f49052a74a6be4 P 9d7ed45c903f4384989599e40b950c9f: stopping tablet replica
I20260928 22:49:02.508584 19164 raft_consensus.cc:2229] T f8fd70d493a6438993f49052a74a6be4 P 9d7ed45c903f4384989599e40b950c9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:49:02.508646 19164 raft_consensus.cc:2258] T f8fd70d493a6438993f49052a74a6be4 P 9d7ed45c903f4384989599e40b950c9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:49:02.508806 19165 tablet_replica.cc:330] T f8fd70d493a6438993f49052a74a6be4 P 5e5a0e809d9842da90487a529361bc7d: stopping tablet replica
I20260928 22:49:02.508903 19165 raft_consensus.cc:2229] T f8fd70d493a6438993f49052a74a6be4 P 5e5a0e809d9842da90487a529361bc7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:49:02.508958 19165 raft_consensus.cc:2258] T f8fd70d493a6438993f49052a74a6be4 P 5e5a0e809d9842da90487a529361bc7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:49:02.509495 19165 ts_tablet_manager.cc:1905] T f8fd70d493a6438993f49052a74a6be4 P 5e5a0e809d9842da90487a529361bc7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:49:02.510710 19163 raft_consensus.cc:2258] T f8fd70d493a6438993f49052a74a6be4 P c9a7dce56d024b8998f3a23395492f1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:49:02.511332 19163 ts_tablet_manager.cc:1905] T f8fd70d493a6438993f49052a74a6be4 P c9a7dce56d024b8998f3a23395492f1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:49:02.511826 19164 ts_tablet_manager.cc:1905] T f8fd70d493a6438993f49052a74a6be4 P 9d7ed45c903f4384989599e40b950c9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:49:02.513391 19163 ts_tablet_manager.cc:1918] T f8fd70d493a6438993f49052a74a6be4 P c9a7dce56d024b8998f3a23395492f1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 22:49:02.513674 19163 log.cc:1196] T f8fd70d493a6438993f49052a74a6be4 P c9a7dce56d024b8998f3a23395492f1c: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-2/wal/wals/f8fd70d493a6438993f49052a74a6be4
I20260928 22:49:02.514062 19163 ts_tablet_manager.cc:1939] T f8fd70d493a6438993f49052a74a6be4 P c9a7dce56d024b8998f3a23395492f1c: Deleting consensus metadata
I20260928 22:49:02.514577 19164 ts_tablet_manager.cc:1918] T f8fd70d493a6438993f49052a74a6be4 P 9d7ed45c903f4384989599e40b950c9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 22:49:02.514598 18597 catalog_manager.cc:4841] TS c9a7dce56d024b8998f3a23395492f1c (127.16.121.131:41721): tablet f8fd70d493a6438993f49052a74a6be4 (table test-table [id=0f972f4ccc2d4459b584bcf38793fef4]) successfully deleted
I20260928 22:49:02.514652 19164 log.cc:1196] T f8fd70d493a6438993f49052a74a6be4 P 9d7ed45c903f4384989599e40b950c9f: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-1/wal/wals/f8fd70d493a6438993f49052a74a6be4
I20260928 22:49:02.515398 19165 ts_tablet_manager.cc:1918] T f8fd70d493a6438993f49052a74a6be4 P 5e5a0e809d9842da90487a529361bc7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 22:49:02.515478 19165 log.cc:1196] T f8fd70d493a6438993f49052a74a6be4 P 5e5a0e809d9842da90487a529361bc7d: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-0/wal/wals/f8fd70d493a6438993f49052a74a6be4
I20260928 22:49:02.515733 19165 ts_tablet_manager.cc:1939] T f8fd70d493a6438993f49052a74a6be4 P 5e5a0e809d9842da90487a529361bc7d: Deleting consensus metadata
I20260928 22:49:02.516136 19164 ts_tablet_manager.cc:1939] T f8fd70d493a6438993f49052a74a6be4 P 9d7ed45c903f4384989599e40b950c9f: Deleting consensus metadata
I20260928 22:49:02.516587 18596 catalog_manager.cc:4841] TS 5e5a0e809d9842da90487a529361bc7d (127.16.121.129:41735): tablet f8fd70d493a6438993f49052a74a6be4 (table test-table [id=0f972f4ccc2d4459b584bcf38793fef4]) successfully deleted
I20260928 22:49:02.516608 18595 catalog_manager.cc:4841] TS 9d7ed45c903f4384989599e40b950c9f (127.16.121.130:38211): tablet f8fd70d493a6438993f49052a74a6be4 (table test-table [id=0f972f4ccc2d4459b584bcf38793fef4]) successfully deleted
I20260928 22:49:02.672161 19124 raft_consensus.cc:482] T f65aaa316d8f41709def1ed95149d69a P 9d7ed45c903f4384989599e40b950c9f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:49:02.672329 19124 raft_consensus.cc:504] T f65aaa316d8f41709def1ed95149d69a P 9d7ed45c903f4384989599e40b950c9f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } }
I20260928 22:49:02.672693 19124 leader_election.cc:288] T f65aaa316d8f41709def1ed95149d69a P 9d7ed45c903f4384989599e40b950c9f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e5a0e809d9842da90487a529361bc7d (127.16.121.129:41735), c9a7dce56d024b8998f3a23395492f1c (127.16.121.131:41721)
I20260928 22:49:02.677129 19115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:02.673016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:50861 (local address 127.16.121.131:41721)
0928 22:49:02.673079 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:02.673081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:02.673168 (+ 87us) server_negotiation.cc:407] Connection header received
0928 22:49:02.673237 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:02.673238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:02.673253 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:02.673323 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:49:02.673612 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.674390 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.676134 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.676687 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.676772 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:02.676783 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:02.677011 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 22:49:02.677055 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 22:49:02.677129 19114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:02.673008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:59015 (local address 127.16.121.129:41735)
0928 22:49:02.673071 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:02.673073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:02.673168 (+ 95us) server_negotiation.cc:407] Connection header received
0928 22:49:02.673258 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:02.673260 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:02.673269 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:02.673323 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:49:02.673639 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.674391 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.676135 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.676747 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.676813 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:02.676819 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:02.677011 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 22:49:02.677051 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 22:49:02.677163 19170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:02.672930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.131:41721 (local address 127.16.121.130:50861)
0928 22:49:02.673094 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:02.673104 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:02.673170 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:02.673353 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:02.673354 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:02.673356 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:49:02.673477 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:02.673480 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.674546 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.674548 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:02.676028 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:02.676031 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.676805 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.676806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:02.676938 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:02.676941 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:49:02.676996 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 22:49:02.677048 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":80,"thread_start_us":94,"threads_started":1}
I20260928 22:49:02.677163 19169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:02.672930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:41735 (local address 127.16.121.130:59015)
0928 22:49:02.673072 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:02.673087 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:02.673170 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:02.673353 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:02.673354 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:02.673366 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:49:02.673534 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:02.673537 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.674548 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.674549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:02.676028 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:02.676031 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.676839 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.676840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:02.676913 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:02.676919 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 22:49:02.676987 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 22:49:02.677048 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":102,"thread_start_us":68,"threads_started":1}
I20260928 22:49:02.677521 19047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f65aaa316d8f41709def1ed95149d69a" candidate_uuid: "9d7ed45c903f4384989599e40b950c9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a7dce56d024b8998f3a23395492f1c" is_pre_election: true
I20260928 22:49:02.677625 19047 raft_consensus.cc:2454] T f65aaa316d8f41709def1ed95149d69a P c9a7dce56d024b8998f3a23395492f1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d7ed45c903f4384989599e40b950c9f in term 0.
I20260928 22:49:02.677834 18841 leader_election.cc:302] T f65aaa316d8f41709def1ed95149d69a P 9d7ed45c903f4384989599e40b950c9f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d7ed45c903f4384989599e40b950c9f, c9a7dce56d024b8998f3a23395492f1c; no voters:
I20260928 22:49:02.677853 18767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f65aaa316d8f41709def1ed95149d69a" candidate_uuid: "9d7ed45c903f4384989599e40b950c9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e5a0e809d9842da90487a529361bc7d" is_pre_election: true
I20260928 22:49:02.677938 18767 raft_consensus.cc:2454] T f65aaa316d8f41709def1ed95149d69a P 5e5a0e809d9842da90487a529361bc7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d7ed45c903f4384989599e40b950c9f in term 0.
I20260928 22:49:02.678020 19124 raft_consensus.cc:2781] T f65aaa316d8f41709def1ed95149d69a P 9d7ed45c903f4384989599e40b950c9f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:49:02.678059 19124 raft_consensus.cc:482] T f65aaa316d8f41709def1ed95149d69a P 9d7ed45c903f4384989599e40b950c9f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:49:02.678078 19124 raft_consensus.cc:3037] T f65aaa316d8f41709def1ed95149d69a P 9d7ed45c903f4384989599e40b950c9f [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:02.678813 19124 raft_consensus.cc:504] T f65aaa316d8f41709def1ed95149d69a P 9d7ed45c903f4384989599e40b950c9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } }
I20260928 22:49:02.678965 19124 leader_election.cc:288] T f65aaa316d8f41709def1ed95149d69a P 9d7ed45c903f4384989599e40b950c9f [CANDIDATE]: Term 1 election: Requested vote from peers 5e5a0e809d9842da90487a529361bc7d (127.16.121.129:41735), c9a7dce56d024b8998f3a23395492f1c (127.16.121.131:41721)
I20260928 22:49:02.679139 18767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f65aaa316d8f41709def1ed95149d69a" candidate_uuid: "9d7ed45c903f4384989599e40b950c9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e5a0e809d9842da90487a529361bc7d"
I20260928 22:49:02.679171 19047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f65aaa316d8f41709def1ed95149d69a" candidate_uuid: "9d7ed45c903f4384989599e40b950c9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a7dce56d024b8998f3a23395492f1c"
I20260928 22:49:02.679247 18767 raft_consensus.cc:3037] T f65aaa316d8f41709def1ed95149d69a P 5e5a0e809d9842da90487a529361bc7d [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:02.679253 19047 raft_consensus.cc:3037] T f65aaa316d8f41709def1ed95149d69a P c9a7dce56d024b8998f3a23395492f1c [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:02.679859 18767 raft_consensus.cc:2454] T f65aaa316d8f41709def1ed95149d69a P 5e5a0e809d9842da90487a529361bc7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d7ed45c903f4384989599e40b950c9f in term 1.
I20260928 22:49:02.680071 18840 leader_election.cc:302] T f65aaa316d8f41709def1ed95149d69a P 9d7ed45c903f4384989599e40b950c9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e5a0e809d9842da90487a529361bc7d, 9d7ed45c903f4384989599e40b950c9f; no voters:
I20260928 22:49:02.680073 19047 raft_consensus.cc:2454] T f65aaa316d8f41709def1ed95149d69a P c9a7dce56d024b8998f3a23395492f1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d7ed45c903f4384989599e40b950c9f in term 1.
I20260928 22:49:02.680168 19124 raft_consensus.cc:2781] T f65aaa316d8f41709def1ed95149d69a P 9d7ed45c903f4384989599e40b950c9f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:49:02.680284 19124 raft_consensus.cc:686] T f65aaa316d8f41709def1ed95149d69a P 9d7ed45c903f4384989599e40b950c9f [term 1 LEADER]: Becoming Leader. State: Replica: 9d7ed45c903f4384989599e40b950c9f, State: Running, Role: LEADER
I20260928 22:49:02.680361 19124 consensus_queue.cc:237] T f65aaa316d8f41709def1ed95149d69a P 9d7ed45c903f4384989599e40b950c9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } }
I20260928 22:49:02.681030 18609 catalog_manager.cc:5495] T f65aaa316d8f41709def1ed95149d69a P 9d7ed45c903f4384989599e40b950c9f reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d7ed45c903f4384989599e40b950c9f (127.16.121.130). New cstate: current_term: 1 leader_uuid: "9d7ed45c903f4384989599e40b950c9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } health_report { overall_health: UNKNOWN } } }
I20260928 22:49:02.693359 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:02.690842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36000 (local address 127.16.121.130:38211)
0928 22:49:02.690922 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:02.690924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:02.690931 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:49:02.690985 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:02.690985 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:02.691007 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:02.691128 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:49:02.691207 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.691687 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.692441 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.692624 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.692681 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:02.692902 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:49:02.693053 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:49:02.693133 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:02.693174 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 22:49:02.693198 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 22:49:02.699748 19175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:02.696497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:41735 (local address 127.16.121.130:50627)
0928 22:49:02.696725 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:02.696739 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:02.696831 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:02.696985 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:02.696986 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:02.696988 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 22:49:02.697061 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:02.697063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.697750 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.697751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:02.698797 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:02.698800 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.699256 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.699257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:02.699325 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:02.699344 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 22:49:02.699555 (+ 211us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 22:49:02.699557 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 22:49:02.699610 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 22:49:02.699657 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":95,"threads_started":1}
I20260928 22:49:02.699723 19114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:02.696584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:50627 (local address 127.16.121.129:41735)
0928 22:49:02.696685 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:02.696687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:02.696812 (+ 125us) server_negotiation.cc:407] Connection header received
0928 22:49:02.696886 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:02.696887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:02.696898 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:02.696955 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:49:02.697137 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.697663 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.698901 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.699184 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.699257 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:02.699438 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:49:02.699483 (+ 45us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:49:02.699554 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:02.699617 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 22:49:02.699640 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 22:49:02.702253 19047 raft_consensus.cc:1261] T f65aaa316d8f41709def1ed95149d69a P c9a7dce56d024b8998f3a23395492f1c [term 1 FOLLOWER]: Refusing update from remote peer 9d7ed45c903f4384989599e40b950c9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:49:02.702317 18767 raft_consensus.cc:1261] T f65aaa316d8f41709def1ed95149d69a P 5e5a0e809d9842da90487a529361bc7d [term 1 FOLLOWER]: Refusing update from remote peer 9d7ed45c903f4384989599e40b950c9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:49:02.702512 19124 consensus_queue.cc:1035] T f65aaa316d8f41709def1ed95149d69a P 9d7ed45c903f4384989599e40b950c9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:49:02.702662 19171 consensus_queue.cc:1035] T f65aaa316d8f41709def1ed95149d69a P 9d7ed45c903f4384989599e40b950c9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:49:02.709832 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:02.707459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:37885 (local address 127.16.121.130:38211)
0928 22:49:02.707502 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:02.707503 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:02.707549 (+ 46us) server_negotiation.cc:407] Connection header received
0928 22:49:02.707636 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:02.707637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:02.707646 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:02.707695 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:49:02.707838 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.708326 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.709055 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.709250 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.709370 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:02.709586 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:49:02.709625 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:49:02.709679 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:02.709737 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 22:49:02.709757 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 22:49:02.709842 19160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:02.707422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.130:38211 (local address 127.16.121.129:37885)
0928 22:49:02.707505 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:02.707516 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:02.707565 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:02.707698 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:02.707699 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:02.707700 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 22:49:02.707765 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:02.707767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.708408 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.708409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:02.708966 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:02.708968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:02.709367 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:02.709368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:02.709498 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:02.709505 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 22:49:02.709680 (+ 175us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 22:49:02.709683 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:49:02.709726 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 22:49:02.709777 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
W20260928 22:49:02.710312 19146 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 22:49:04.240026 18608 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:49:04.240152 18608 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:49:04.241583 18608 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 622784bfe8d5446abd97cdd40c6d8115: Sending DeleteTablet for 3 replicas of tablet f65aaa316d8f41709def1ed95149d69a
I20260928 22:49:04.241942 18887 tablet_service.cc:1514] Processing DeleteTablet for tablet f65aaa316d8f41709def1ed95149d69a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:49:04 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:35994
I20260928 22:49:04.241953 19027 tablet_service.cc:1514] Processing DeleteTablet for tablet f65aaa316d8f41709def1ed95149d69a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:49:04 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:53014
I20260928 22:49:04.242249 19204 tablet_replica.cc:330] T f65aaa316d8f41709def1ed95149d69a P 9d7ed45c903f4384989599e40b950c9f: stopping tablet replica
I20260928 22:49:04.242328 19203 tablet_replica.cc:330] T f65aaa316d8f41709def1ed95149d69a P c9a7dce56d024b8998f3a23395492f1c: stopping tablet replica
I20260928 22:49:04.242354 19204 raft_consensus.cc:2229] T f65aaa316d8f41709def1ed95149d69a P 9d7ed45c903f4384989599e40b950c9f [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:49:04.242408 19203 raft_consensus.cc:2229] T f65aaa316d8f41709def1ed95149d69a P c9a7dce56d024b8998f3a23395492f1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:49:04.242463 19203 raft_consensus.cc:2258] T f65aaa316d8f41709def1ed95149d69a P c9a7dce56d024b8998f3a23395492f1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:49:04.242587 19204 raft_consensus.cc:2258] T f65aaa316d8f41709def1ed95149d69a P 9d7ed45c903f4384989599e40b950c9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:49:04.242748 19203 ts_tablet_manager.cc:1905] T f65aaa316d8f41709def1ed95149d69a P c9a7dce56d024b8998f3a23395492f1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:49:04.242900 19204 ts_tablet_manager.cc:1905] T f65aaa316d8f41709def1ed95149d69a P 9d7ed45c903f4384989599e40b950c9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:49:04.244778 18747 tablet_service.cc:1514] Processing DeleteTablet for tablet f65aaa316d8f41709def1ed95149d69a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:49:04 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:33930
I20260928 22:49:04.245003 19203 ts_tablet_manager.cc:1918] T f65aaa316d8f41709def1ed95149d69a P c9a7dce56d024b8998f3a23395492f1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 22:49:04.245075 19203 log.cc:1196] T f65aaa316d8f41709def1ed95149d69a P c9a7dce56d024b8998f3a23395492f1c: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-2/wal/wals/f65aaa316d8f41709def1ed95149d69a
I20260928 22:49:04.245043 19204 ts_tablet_manager.cc:1918] T f65aaa316d8f41709def1ed95149d69a P 9d7ed45c903f4384989599e40b950c9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 22:49:04.245182 19204 log.cc:1196] T f65aaa316d8f41709def1ed95149d69a P 9d7ed45c903f4384989599e40b950c9f: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-1/wal/wals/f65aaa316d8f41709def1ed95149d69a
I20260928 22:49:04.245070 19206 tablet_replica.cc:330] T f65aaa316d8f41709def1ed95149d69a P 5e5a0e809d9842da90487a529361bc7d: stopping tablet replica
I20260928 22:49:04.245301 19203 ts_tablet_manager.cc:1939] T f65aaa316d8f41709def1ed95149d69a P c9a7dce56d024b8998f3a23395492f1c: Deleting consensus metadata
I20260928 22:49:04.245283 19206 raft_consensus.cc:2229] T f65aaa316d8f41709def1ed95149d69a P 5e5a0e809d9842da90487a529361bc7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:49:04.245383 19206 raft_consensus.cc:2258] T f65aaa316d8f41709def1ed95149d69a P 5e5a0e809d9842da90487a529361bc7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:49:04.245585 19204 ts_tablet_manager.cc:1939] T f65aaa316d8f41709def1ed95149d69a P 9d7ed45c903f4384989599e40b950c9f: Deleting consensus metadata
I20260928 22:49:04.245869 18597 catalog_manager.cc:4841] TS c9a7dce56d024b8998f3a23395492f1c (127.16.121.131:41721): tablet f65aaa316d8f41709def1ed95149d69a (table test-table [id=f1e4ef48f49d47309f11c7875f6f7b46]) successfully deleted
I20260928 22:49:04.246016 19206 ts_tablet_manager.cc:1905] T f65aaa316d8f41709def1ed95149d69a P 5e5a0e809d9842da90487a529361bc7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:49:04.245958 18595 catalog_manager.cc:4841] TS 9d7ed45c903f4384989599e40b950c9f (127.16.121.130:38211): tablet f65aaa316d8f41709def1ed95149d69a (table test-table [id=f1e4ef48f49d47309f11c7875f6f7b46]) successfully deleted
I20260928 22:49:04.247679 19206 ts_tablet_manager.cc:1918] T f65aaa316d8f41709def1ed95149d69a P 5e5a0e809d9842da90487a529361bc7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 22:49:04.247762 19206 log.cc:1196] T f65aaa316d8f41709def1ed95149d69a P 5e5a0e809d9842da90487a529361bc7d: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-0/wal/wals/f65aaa316d8f41709def1ed95149d69a
I20260928 22:49:04.248030 19206 ts_tablet_manager.cc:1939] T f65aaa316d8f41709def1ed95149d69a P 5e5a0e809d9842da90487a529361bc7d: Deleting consensus metadata
I20260928 22:49:04.248521 18596 catalog_manager.cc:4841] TS 5e5a0e809d9842da90487a529361bc7d (127.16.121.129:41735): tablet f65aaa316d8f41709def1ed95149d69a (table test-table [id=f1e4ef48f49d47309f11c7875f6f7b46]) successfully deleted
Sep 28 22:49:04 dist-test-slave-56vw krb5kdc[18565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635744, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 22:49:04 dist-test-slave-56vw krb5kdc[18565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635744, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:49:04.277500 19213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:04.260718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54442 (local address 127.16.121.190:40849)
0928 22:49:04.260892 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:04.260895 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:04.260907 (+ 12us) server_negotiation.cc:407] Connection header received
0928 22:49:04.260956 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:04.260956 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:04.260966 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:04.261040 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:04.261668 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:04.262243 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:04.263269 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:04.263577 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:04.263678 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:04.274360 (+ 10682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:04.274363 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:04.274363 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:04.274372 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:04.276156 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:04.276585 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:04.276586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:04.276587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:04.276652 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:04.276845 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:04.276845 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:04.276845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:04.276986 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:04.277066 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:04.277182 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 22:49:04.277271 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":84,"threads_started":1}
I20260928 22:49:04.279510 18608 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54442:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:49:04.279673 18608 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:49:04.281417 18887 tablet_service.cc:1467] Processing CreateTablet for tablet 4fdec6916e11419bbd00268b443e5469 (DEFAULT_TABLE table=test-table [id=0e8915f17ffd4b26a107cf26c6857326]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:49:04.281448 19027 tablet_service.cc:1467] Processing CreateTablet for tablet 4fdec6916e11419bbd00268b443e5469 (DEFAULT_TABLE table=test-table [id=0e8915f17ffd4b26a107cf26c6857326]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:49:04.281585 18887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fdec6916e11419bbd00268b443e5469. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:04.281605 19027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fdec6916e11419bbd00268b443e5469. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:04.282706 19214 tablet_bootstrap.cc:492] T 4fdec6916e11419bbd00268b443e5469 P c9a7dce56d024b8998f3a23395492f1c: Bootstrap starting.
I20260928 22:49:04.283097 19215 tablet_bootstrap.cc:492] T 4fdec6916e11419bbd00268b443e5469 P 9d7ed45c903f4384989599e40b950c9f: Bootstrap starting.
I20260928 22:49:04.283200 19214 tablet_bootstrap.cc:654] T 4fdec6916e11419bbd00268b443e5469 P c9a7dce56d024b8998f3a23395492f1c: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:04.283185 18747 tablet_service.cc:1467] Processing CreateTablet for tablet 4fdec6916e11419bbd00268b443e5469 (DEFAULT_TABLE table=test-table [id=0e8915f17ffd4b26a107cf26c6857326]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:49:04.283313 18747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fdec6916e11419bbd00268b443e5469. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:04.283509 19215 tablet_bootstrap.cc:654] T 4fdec6916e11419bbd00268b443e5469 P 9d7ed45c903f4384989599e40b950c9f: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:04.283833 19214 tablet_bootstrap.cc:492] T 4fdec6916e11419bbd00268b443e5469 P c9a7dce56d024b8998f3a23395492f1c: No bootstrap required, opened a new log
I20260928 22:49:04.283893 19214 ts_tablet_manager.cc:1397] T 4fdec6916e11419bbd00268b443e5469 P c9a7dce56d024b8998f3a23395492f1c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 22:49:04.284019 19214 raft_consensus.cc:348] T 4fdec6916e11419bbd00268b443e5469 P c9a7dce56d024b8998f3a23395492f1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } }
I20260928 22:49:04.284113 19214 raft_consensus.cc:374] T 4fdec6916e11419bbd00268b443e5469 P c9a7dce56d024b8998f3a23395492f1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:04.284125 19215 tablet_bootstrap.cc:492] T 4fdec6916e11419bbd00268b443e5469 P 9d7ed45c903f4384989599e40b950c9f: No bootstrap required, opened a new log
I20260928 22:49:04.284138 19214 raft_consensus.cc:729] T 4fdec6916e11419bbd00268b443e5469 P c9a7dce56d024b8998f3a23395492f1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9a7dce56d024b8998f3a23395492f1c, State: Initialized, Role: FOLLOWER
I20260928 22:49:04.284183 19215 ts_tablet_manager.cc:1397] T 4fdec6916e11419bbd00268b443e5469 P 9d7ed45c903f4384989599e40b950c9f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:04.284210 19214 consensus_queue.cc:260] T 4fdec6916e11419bbd00268b443e5469 P c9a7dce56d024b8998f3a23395492f1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } }
I20260928 22:49:04.284387 19214 ts_tablet_manager.cc:1428] T 4fdec6916e11419bbd00268b443e5469 P c9a7dce56d024b8998f3a23395492f1c: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 22:49:04.284476 19215 raft_consensus.cc:348] T 4fdec6916e11419bbd00268b443e5469 P 9d7ed45c903f4384989599e40b950c9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } }
I20260928 22:49:04.284549 19215 raft_consensus.cc:374] T 4fdec6916e11419bbd00268b443e5469 P 9d7ed45c903f4384989599e40b950c9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:04.284575 19215 raft_consensus.cc:729] T 4fdec6916e11419bbd00268b443e5469 P 9d7ed45c903f4384989599e40b950c9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7ed45c903f4384989599e40b950c9f, State: Initialized, Role: FOLLOWER
I20260928 22:49:04.284624 19215 consensus_queue.cc:260] T 4fdec6916e11419bbd00268b443e5469 P 9d7ed45c903f4384989599e40b950c9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } }
I20260928 22:49:04.284796 19215 ts_tablet_manager.cc:1428] T 4fdec6916e11419bbd00268b443e5469 P 9d7ed45c903f4384989599e40b950c9f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:04.284847 19219 tablet_bootstrap.cc:492] T 4fdec6916e11419bbd00268b443e5469 P 5e5a0e809d9842da90487a529361bc7d: Bootstrap starting.
I20260928 22:49:04.285446 19219 tablet_bootstrap.cc:654] T 4fdec6916e11419bbd00268b443e5469 P 5e5a0e809d9842da90487a529361bc7d: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:04.285983 19219 tablet_bootstrap.cc:492] T 4fdec6916e11419bbd00268b443e5469 P 5e5a0e809d9842da90487a529361bc7d: No bootstrap required, opened a new log
I20260928 22:49:04.286051 19219 ts_tablet_manager.cc:1397] T 4fdec6916e11419bbd00268b443e5469 P 5e5a0e809d9842da90487a529361bc7d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:04.286172 19219 raft_consensus.cc:348] T 4fdec6916e11419bbd00268b443e5469 P 5e5a0e809d9842da90487a529361bc7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } }
I20260928 22:49:04.286247 19219 raft_consensus.cc:374] T 4fdec6916e11419bbd00268b443e5469 P 5e5a0e809d9842da90487a529361bc7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:04.286270 19219 raft_consensus.cc:729] T 4fdec6916e11419bbd00268b443e5469 P 5e5a0e809d9842da90487a529361bc7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e5a0e809d9842da90487a529361bc7d, State: Initialized, Role: FOLLOWER
I20260928 22:49:04.286327 19219 consensus_queue.cc:260] T 4fdec6916e11419bbd00268b443e5469 P 5e5a0e809d9842da90487a529361bc7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } }
I20260928 22:49:04.286419 19219 ts_tablet_manager.cc:1428] T 4fdec6916e11419bbd00268b443e5469 P 5e5a0e809d9842da90487a529361bc7d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:04.297006 19196 raft_consensus.cc:482] T 4fdec6916e11419bbd00268b443e5469 P 9d7ed45c903f4384989599e40b950c9f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:49:04.297106 19196 raft_consensus.cc:504] T 4fdec6916e11419bbd00268b443e5469 P 9d7ed45c903f4384989599e40b950c9f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } }
I20260928 22:49:04.297281 19196 leader_election.cc:288] T 4fdec6916e11419bbd00268b443e5469 P 9d7ed45c903f4384989599e40b950c9f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e5a0e809d9842da90487a529361bc7d (127.16.121.129:41735), c9a7dce56d024b8998f3a23395492f1c (127.16.121.131:41721)
I20260928 22:49:04.297497 18767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fdec6916e11419bbd00268b443e5469" candidate_uuid: "9d7ed45c903f4384989599e40b950c9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e5a0e809d9842da90487a529361bc7d" is_pre_election: true
I20260928 22:49:04.297622 18767 raft_consensus.cc:2454] T 4fdec6916e11419bbd00268b443e5469 P 5e5a0e809d9842da90487a529361bc7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d7ed45c903f4384989599e40b950c9f in term 0.
I20260928 22:49:04.297637 19047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fdec6916e11419bbd00268b443e5469" candidate_uuid: "9d7ed45c903f4384989599e40b950c9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a7dce56d024b8998f3a23395492f1c" is_pre_election: true
I20260928 22:49:04.297722 19047 raft_consensus.cc:2454] T 4fdec6916e11419bbd00268b443e5469 P c9a7dce56d024b8998f3a23395492f1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d7ed45c903f4384989599e40b950c9f in term 0.
I20260928 22:49:04.297902 18841 leader_election.cc:302] T 4fdec6916e11419bbd00268b443e5469 P 9d7ed45c903f4384989599e40b950c9f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5e5a0e809d9842da90487a529361bc7d, 9d7ed45c903f4384989599e40b950c9f, c9a7dce56d024b8998f3a23395492f1c; no voters:
I20260928 22:49:04.298007 19196 raft_consensus.cc:2781] T 4fdec6916e11419bbd00268b443e5469 P 9d7ed45c903f4384989599e40b950c9f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:49:04.298045 19196 raft_consensus.cc:482] T 4fdec6916e11419bbd00268b443e5469 P 9d7ed45c903f4384989599e40b950c9f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:49:04.298074 19196 raft_consensus.cc:3037] T 4fdec6916e11419bbd00268b443e5469 P 9d7ed45c903f4384989599e40b950c9f [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:04.298800 19196 raft_consensus.cc:504] T 4fdec6916e11419bbd00268b443e5469 P 9d7ed45c903f4384989599e40b950c9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } }
I20260928 22:49:04.298979 19196 leader_election.cc:288] T 4fdec6916e11419bbd00268b443e5469 P 9d7ed45c903f4384989599e40b950c9f [CANDIDATE]: Term 1 election: Requested vote from peers 5e5a0e809d9842da90487a529361bc7d (127.16.121.129:41735), c9a7dce56d024b8998f3a23395492f1c (127.16.121.131:41721)
I20260928 22:49:04.299139 18767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fdec6916e11419bbd00268b443e5469" candidate_uuid: "9d7ed45c903f4384989599e40b950c9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e5a0e809d9842da90487a529361bc7d"
I20260928 22:49:04.299173 19047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fdec6916e11419bbd00268b443e5469" candidate_uuid: "9d7ed45c903f4384989599e40b950c9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a7dce56d024b8998f3a23395492f1c"
I20260928 22:49:04.299256 18767 raft_consensus.cc:3037] T 4fdec6916e11419bbd00268b443e5469 P 5e5a0e809d9842da90487a529361bc7d [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:04.299254 19047 raft_consensus.cc:3037] T 4fdec6916e11419bbd00268b443e5469 P c9a7dce56d024b8998f3a23395492f1c [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:04.299863 19047 raft_consensus.cc:2454] T 4fdec6916e11419bbd00268b443e5469 P c9a7dce56d024b8998f3a23395492f1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d7ed45c903f4384989599e40b950c9f in term 1.
I20260928 22:49:04.300022 18767 raft_consensus.cc:2454] T 4fdec6916e11419bbd00268b443e5469 P 5e5a0e809d9842da90487a529361bc7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d7ed45c903f4384989599e40b950c9f in term 1.
I20260928 22:49:04.300060 18841 leader_election.cc:302] T 4fdec6916e11419bbd00268b443e5469 P 9d7ed45c903f4384989599e40b950c9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d7ed45c903f4384989599e40b950c9f, c9a7dce56d024b8998f3a23395492f1c; no voters:
I20260928 22:49:04.300171 19196 raft_consensus.cc:2781] T 4fdec6916e11419bbd00268b443e5469 P 9d7ed45c903f4384989599e40b950c9f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:49:04.300225 19196 raft_consensus.cc:686] T 4fdec6916e11419bbd00268b443e5469 P 9d7ed45c903f4384989599e40b950c9f [term 1 LEADER]: Becoming Leader. State: Replica: 9d7ed45c903f4384989599e40b950c9f, State: Running, Role: LEADER
I20260928 22:49:04.300284 19196 consensus_queue.cc:237] T 4fdec6916e11419bbd00268b443e5469 P 9d7ed45c903f4384989599e40b950c9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } }
I20260928 22:49:04.300935 18610 catalog_manager.cc:5495] T 4fdec6916e11419bbd00268b443e5469 P 9d7ed45c903f4384989599e40b950c9f reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d7ed45c903f4384989599e40b950c9f (127.16.121.130). New cstate: current_term: 1 leader_uuid: "9d7ed45c903f4384989599e40b950c9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } health_report { overall_health: UNKNOWN } } }
I20260928 22:49:04.311203 19223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:04.308256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36006 (local address 127.16.121.130:38211)
0928 22:49:04.308494 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:04.308496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:04.308509 (+ 13us) server_negotiation.cc:407] Connection header received
0928 22:49:04.308531 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:04.308531 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:04.308543 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:04.308617 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:49:04.308828 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:04.309453 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:04.310237 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:04.310594 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:04.310662 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:04.310829 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:49:04.310867 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:49:04.310914 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:04.311014 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 22:49:04.311034 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":124,"threads_started":1}
I20260928 22:49:04.313982 19047 raft_consensus.cc:1261] T 4fdec6916e11419bbd00268b443e5469 P c9a7dce56d024b8998f3a23395492f1c [term 1 FOLLOWER]: Refusing update from remote peer 9d7ed45c903f4384989599e40b950c9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:49:04.313975 18767 raft_consensus.cc:1261] T 4fdec6916e11419bbd00268b443e5469 P 5e5a0e809d9842da90487a529361bc7d [term 1 FOLLOWER]: Refusing update from remote peer 9d7ed45c903f4384989599e40b950c9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:49:04.314198 19196 consensus_queue.cc:1035] T 4fdec6916e11419bbd00268b443e5469 P 9d7ed45c903f4384989599e40b950c9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:49:04.314332 19192 consensus_queue.cc:1035] T 4fdec6916e11419bbd00268b443e5469 P 9d7ed45c903f4384989599e40b950c9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:49:04.324363 18608 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:49:04.324482 18608 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:49:04.325197 18608 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 622784bfe8d5446abd97cdd40c6d8115: Sending DeleteTablet for 3 replicas of tablet 4fdec6916e11419bbd00268b443e5469
I20260928 22:49:04.325472 18747 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fdec6916e11419bbd00268b443e5469 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:49:04 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:33930
I20260928 22:49:04.325532 19027 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fdec6916e11419bbd00268b443e5469 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:49:04 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:53014
I20260928 22:49:04.325594 19206 tablet_replica.cc:330] T 4fdec6916e11419bbd00268b443e5469 P 5e5a0e809d9842da90487a529361bc7d: stopping tablet replica
I20260928 22:49:04.325641 19203 tablet_replica.cc:330] T 4fdec6916e11419bbd00268b443e5469 P c9a7dce56d024b8998f3a23395492f1c: stopping tablet replica
I20260928 22:49:04.325663 19206 raft_consensus.cc:2229] T 4fdec6916e11419bbd00268b443e5469 P 5e5a0e809d9842da90487a529361bc7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:49:04.325711 19206 raft_consensus.cc:2258] T 4fdec6916e11419bbd00268b443e5469 P 5e5a0e809d9842da90487a529361bc7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:49:04.325712 19203 raft_consensus.cc:2229] T 4fdec6916e11419bbd00268b443e5469 P c9a7dce56d024b8998f3a23395492f1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:49:04.325696 18887 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fdec6916e11419bbd00268b443e5469 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:49:04 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:35994
I20260928 22:49:04.325760 19203 raft_consensus.cc:2258] T 4fdec6916e11419bbd00268b443e5469 P c9a7dce56d024b8998f3a23395492f1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:49:04.325778 19204 tablet_replica.cc:330] T 4fdec6916e11419bbd00268b443e5469 P 9d7ed45c903f4384989599e40b950c9f: stopping tablet replica
I20260928 22:49:04.325829 19204 raft_consensus.cc:2229] T 4fdec6916e11419bbd00268b443e5469 P 9d7ed45c903f4384989599e40b950c9f [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:49:04.325933 19204 raft_consensus.cc:2258] T 4fdec6916e11419bbd00268b443e5469 P 9d7ed45c903f4384989599e40b950c9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:49:04.326184 19206 ts_tablet_manager.cc:1905] T 4fdec6916e11419bbd00268b443e5469 P 5e5a0e809d9842da90487a529361bc7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:49:04.326274 19204 ts_tablet_manager.cc:1905] T 4fdec6916e11419bbd00268b443e5469 P 9d7ed45c903f4384989599e40b950c9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:49:04.326366 19203 ts_tablet_manager.cc:1905] T 4fdec6916e11419bbd00268b443e5469 P c9a7dce56d024b8998f3a23395492f1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:49:04.327171 18608 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54442:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:49:04.327330 18608 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:49:04.328063 19204 ts_tablet_manager.cc:1918] T 4fdec6916e11419bbd00268b443e5469 P 9d7ed45c903f4384989599e40b950c9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 22:49:04.328080 19206 ts_tablet_manager.cc:1918] T 4fdec6916e11419bbd00268b443e5469 P 5e5a0e809d9842da90487a529361bc7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 22:49:04.328132 19204 log.cc:1196] T 4fdec6916e11419bbd00268b443e5469 P 9d7ed45c903f4384989599e40b950c9f: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-1/wal/wals/4fdec6916e11419bbd00268b443e5469
I20260928 22:49:04.328138 19206 log.cc:1196] T 4fdec6916e11419bbd00268b443e5469 P 5e5a0e809d9842da90487a529361bc7d: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-0/wal/wals/4fdec6916e11419bbd00268b443e5469
I20260928 22:49:04.328186 19203 ts_tablet_manager.cc:1918] T 4fdec6916e11419bbd00268b443e5469 P c9a7dce56d024b8998f3a23395492f1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 22:49:04.328244 19203 log.cc:1196] T 4fdec6916e11419bbd00268b443e5469 P c9a7dce56d024b8998f3a23395492f1c: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-2/wal/wals/4fdec6916e11419bbd00268b443e5469
I20260928 22:49:04.328380 19206 ts_tablet_manager.cc:1939] T 4fdec6916e11419bbd00268b443e5469 P 5e5a0e809d9842da90487a529361bc7d: Deleting consensus metadata
I20260928 22:49:04.328385 19204 ts_tablet_manager.cc:1939] T 4fdec6916e11419bbd00268b443e5469 P 9d7ed45c903f4384989599e40b950c9f: Deleting consensus metadata
I20260928 22:49:04.328665 19203 ts_tablet_manager.cc:1939] T 4fdec6916e11419bbd00268b443e5469 P c9a7dce56d024b8998f3a23395492f1c: Deleting consensus metadata
I20260928 22:49:04.328881 18596 catalog_manager.cc:4841] TS 5e5a0e809d9842da90487a529361bc7d (127.16.121.129:41735): tablet 4fdec6916e11419bbd00268b443e5469 (table test-table [id=0e8915f17ffd4b26a107cf26c6857326]) successfully deleted
I20260928 22:49:04.329034 18597 catalog_manager.cc:4841] TS c9a7dce56d024b8998f3a23395492f1c (127.16.121.131:41721): tablet 4fdec6916e11419bbd00268b443e5469 (table test-table [id=0e8915f17ffd4b26a107cf26c6857326]) successfully deleted
I20260928 22:49:04.329147 18595 catalog_manager.cc:4841] TS 9d7ed45c903f4384989599e40b950c9f (127.16.121.130:38211): tablet 4fdec6916e11419bbd00268b443e5469 (table test-table [id=0e8915f17ffd4b26a107cf26c6857326]) successfully deleted
I20260928 22:49:04.330096 18887 tablet_service.cc:1467] Processing CreateTablet for tablet 5cfe9fdf0db749649d0651b47ccbd98c (DEFAULT_TABLE table=test-table [id=1bce01434c1c4b7dbbd744968549c2f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:49:04.330152 19027 tablet_service.cc:1467] Processing CreateTablet for tablet 5cfe9fdf0db749649d0651b47ccbd98c (DEFAULT_TABLE table=test-table [id=1bce01434c1c4b7dbbd744968549c2f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:49:04.330152 18747 tablet_service.cc:1467] Processing CreateTablet for tablet 5cfe9fdf0db749649d0651b47ccbd98c (DEFAULT_TABLE table=test-table [id=1bce01434c1c4b7dbbd744968549c2f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:49:04.330246 18887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cfe9fdf0db749649d0651b47ccbd98c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:04.330256 19027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cfe9fdf0db749649d0651b47ccbd98c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:04.330256 18747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cfe9fdf0db749649d0651b47ccbd98c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:04.331506 19215 tablet_bootstrap.cc:492] T 5cfe9fdf0db749649d0651b47ccbd98c P 9d7ed45c903f4384989599e40b950c9f: Bootstrap starting.
I20260928 22:49:04.331538 19214 tablet_bootstrap.cc:492] T 5cfe9fdf0db749649d0651b47ccbd98c P c9a7dce56d024b8998f3a23395492f1c: Bootstrap starting.
I20260928 22:49:04.331840 19215 tablet_bootstrap.cc:654] T 5cfe9fdf0db749649d0651b47ccbd98c P 9d7ed45c903f4384989599e40b950c9f: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:04.331849 19214 tablet_bootstrap.cc:654] T 5cfe9fdf0db749649d0651b47ccbd98c P c9a7dce56d024b8998f3a23395492f1c: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:04.332177 19219 tablet_bootstrap.cc:492] T 5cfe9fdf0db749649d0651b47ccbd98c P 5e5a0e809d9842da90487a529361bc7d: Bootstrap starting.
I20260928 22:49:04.332517 19215 tablet_bootstrap.cc:492] T 5cfe9fdf0db749649d0651b47ccbd98c P 9d7ed45c903f4384989599e40b950c9f: No bootstrap required, opened a new log
I20260928 22:49:04.332528 19214 tablet_bootstrap.cc:492] T 5cfe9fdf0db749649d0651b47ccbd98c P c9a7dce56d024b8998f3a23395492f1c: No bootstrap required, opened a new log
I20260928 22:49:04.332571 19219 tablet_bootstrap.cc:654] T 5cfe9fdf0db749649d0651b47ccbd98c P 5e5a0e809d9842da90487a529361bc7d: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:04.332574 19214 ts_tablet_manager.cc:1397] T 5cfe9fdf0db749649d0651b47ccbd98c P c9a7dce56d024b8998f3a23395492f1c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 22:49:04.332587 19215 ts_tablet_manager.cc:1397] T 5cfe9fdf0db749649d0651b47ccbd98c P 9d7ed45c903f4384989599e40b950c9f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:04.332697 19215 raft_consensus.cc:348] T 5cfe9fdf0db749649d0651b47ccbd98c P 9d7ed45c903f4384989599e40b950c9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } }
I20260928 22:49:04.332690 19214 raft_consensus.cc:348] T 5cfe9fdf0db749649d0651b47ccbd98c P c9a7dce56d024b8998f3a23395492f1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } }
I20260928 22:49:04.332785 19215 raft_consensus.cc:374] T 5cfe9fdf0db749649d0651b47ccbd98c P 9d7ed45c903f4384989599e40b950c9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:04.332787 19214 raft_consensus.cc:374] T 5cfe9fdf0db749649d0651b47ccbd98c P c9a7dce56d024b8998f3a23395492f1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:04.332814 19215 raft_consensus.cc:729] T 5cfe9fdf0db749649d0651b47ccbd98c P 9d7ed45c903f4384989599e40b950c9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7ed45c903f4384989599e40b950c9f, State: Initialized, Role: FOLLOWER
I20260928 22:49:04.332815 19214 raft_consensus.cc:729] T 5cfe9fdf0db749649d0651b47ccbd98c P c9a7dce56d024b8998f3a23395492f1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9a7dce56d024b8998f3a23395492f1c, State: Initialized, Role: FOLLOWER
I20260928 22:49:04.332856 19215 consensus_queue.cc:260] T 5cfe9fdf0db749649d0651b47ccbd98c P 9d7ed45c903f4384989599e40b950c9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } }
I20260928 22:49:04.332901 19214 consensus_queue.cc:260] T 5cfe9fdf0db749649d0651b47ccbd98c P c9a7dce56d024b8998f3a23395492f1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } }
I20260928 22:49:04.332953 19215 ts_tablet_manager.cc:1428] T 5cfe9fdf0db749649d0651b47ccbd98c P 9d7ed45c903f4384989599e40b950c9f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 22:49:04.333061 19219 tablet_bootstrap.cc:492] T 5cfe9fdf0db749649d0651b47ccbd98c P 5e5a0e809d9842da90487a529361bc7d: No bootstrap required, opened a new log
I20260928 22:49:04.333112 19219 ts_tablet_manager.cc:1397] T 5cfe9fdf0db749649d0651b47ccbd98c P 5e5a0e809d9842da90487a529361bc7d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:04.333128 19214 ts_tablet_manager.cc:1428] T 5cfe9fdf0db749649d0651b47ccbd98c P c9a7dce56d024b8998f3a23395492f1c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 22:49:04.333209 19219 raft_consensus.cc:348] T 5cfe9fdf0db749649d0651b47ccbd98c P 5e5a0e809d9842da90487a529361bc7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } }
I20260928 22:49:04.333269 19219 raft_consensus.cc:374] T 5cfe9fdf0db749649d0651b47ccbd98c P 5e5a0e809d9842da90487a529361bc7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:04.333282 19219 raft_consensus.cc:729] T 5cfe9fdf0db749649d0651b47ccbd98c P 5e5a0e809d9842da90487a529361bc7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e5a0e809d9842da90487a529361bc7d, State: Initialized, Role: FOLLOWER
I20260928 22:49:04.333346 19219 consensus_queue.cc:260] T 5cfe9fdf0db749649d0651b47ccbd98c P 5e5a0e809d9842da90487a529361bc7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } }
I20260928 22:49:04.333432 19219 ts_tablet_manager.cc:1428] T 5cfe9fdf0db749649d0651b47ccbd98c P 5e5a0e809d9842da90487a529361bc7d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:04.461941 19139 raft_consensus.cc:482] T 5cfe9fdf0db749649d0651b47ccbd98c P 5e5a0e809d9842da90487a529361bc7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:49:04.462061 19139 raft_consensus.cc:504] T 5cfe9fdf0db749649d0651b47ccbd98c P 5e5a0e809d9842da90487a529361bc7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } }
I20260928 22:49:04.462327 19139 leader_election.cc:288] T 5cfe9fdf0db749649d0651b47ccbd98c P 5e5a0e809d9842da90487a529361bc7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d7ed45c903f4384989599e40b950c9f (127.16.121.130:38211), c9a7dce56d024b8998f3a23395492f1c (127.16.121.131:41721)
I20260928 22:49:04.462669 18907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cfe9fdf0db749649d0651b47ccbd98c" candidate_uuid: "5e5a0e809d9842da90487a529361bc7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7ed45c903f4384989599e40b950c9f" is_pre_election: true
I20260928 22:49:04.462663 19047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cfe9fdf0db749649d0651b47ccbd98c" candidate_uuid: "5e5a0e809d9842da90487a529361bc7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a7dce56d024b8998f3a23395492f1c" is_pre_election: true
I20260928 22:49:04.462776 18907 raft_consensus.cc:2454] T 5cfe9fdf0db749649d0651b47ccbd98c P 9d7ed45c903f4384989599e40b950c9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e5a0e809d9842da90487a529361bc7d in term 0.
I20260928 22:49:04.462776 19047 raft_consensus.cc:2454] T 5cfe9fdf0db749649d0651b47ccbd98c P c9a7dce56d024b8998f3a23395492f1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e5a0e809d9842da90487a529361bc7d in term 0.
I20260928 22:49:04.463019 18699 leader_election.cc:302] T 5cfe9fdf0db749649d0651b47ccbd98c P 5e5a0e809d9842da90487a529361bc7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e5a0e809d9842da90487a529361bc7d, 9d7ed45c903f4384989599e40b950c9f; no voters:
I20260928 22:49:04.463212 19139 raft_consensus.cc:2781] T 5cfe9fdf0db749649d0651b47ccbd98c P 5e5a0e809d9842da90487a529361bc7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:49:04.463284 19139 raft_consensus.cc:482] T 5cfe9fdf0db749649d0651b47ccbd98c P 5e5a0e809d9842da90487a529361bc7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:49:04.463316 19139 raft_consensus.cc:3037] T 5cfe9fdf0db749649d0651b47ccbd98c P 5e5a0e809d9842da90487a529361bc7d [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:04.464066 19139 raft_consensus.cc:504] T 5cfe9fdf0db749649d0651b47ccbd98c P 5e5a0e809d9842da90487a529361bc7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } }
I20260928 22:49:04.464226 19139 leader_election.cc:288] T 5cfe9fdf0db749649d0651b47ccbd98c P 5e5a0e809d9842da90487a529361bc7d [CANDIDATE]: Term 1 election: Requested vote from peers 9d7ed45c903f4384989599e40b950c9f (127.16.121.130:38211), c9a7dce56d024b8998f3a23395492f1c (127.16.121.131:41721)
I20260928 22:49:04.464403 18907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cfe9fdf0db749649d0651b47ccbd98c" candidate_uuid: "5e5a0e809d9842da90487a529361bc7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d7ed45c903f4384989599e40b950c9f"
I20260928 22:49:04.464391 19047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cfe9fdf0db749649d0651b47ccbd98c" candidate_uuid: "5e5a0e809d9842da90487a529361bc7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a7dce56d024b8998f3a23395492f1c"
I20260928 22:49:04.464478 19047 raft_consensus.cc:3037] T 5cfe9fdf0db749649d0651b47ccbd98c P c9a7dce56d024b8998f3a23395492f1c [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:04.464478 18907 raft_consensus.cc:3037] T 5cfe9fdf0db749649d0651b47ccbd98c P 9d7ed45c903f4384989599e40b950c9f [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:04.465261 18907 raft_consensus.cc:2454] T 5cfe9fdf0db749649d0651b47ccbd98c P 9d7ed45c903f4384989599e40b950c9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e5a0e809d9842da90487a529361bc7d in term 1.
I20260928 22:49:04.465340 19047 raft_consensus.cc:2454] T 5cfe9fdf0db749649d0651b47ccbd98c P c9a7dce56d024b8998f3a23395492f1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e5a0e809d9842da90487a529361bc7d in term 1.
I20260928 22:49:04.465498 18699 leader_election.cc:302] T 5cfe9fdf0db749649d0651b47ccbd98c P 5e5a0e809d9842da90487a529361bc7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e5a0e809d9842da90487a529361bc7d, 9d7ed45c903f4384989599e40b950c9f; no voters:
I20260928 22:49:04.465593 19139 raft_consensus.cc:2781] T 5cfe9fdf0db749649d0651b47ccbd98c P 5e5a0e809d9842da90487a529361bc7d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:49:04.465684 19139 raft_consensus.cc:686] T 5cfe9fdf0db749649d0651b47ccbd98c P 5e5a0e809d9842da90487a529361bc7d [term 1 LEADER]: Becoming Leader. State: Replica: 5e5a0e809d9842da90487a529361bc7d, State: Running, Role: LEADER
I20260928 22:49:04.465742 19139 consensus_queue.cc:237] T 5cfe9fdf0db749649d0651b47ccbd98c P 5e5a0e809d9842da90487a529361bc7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } }
I20260928 22:49:04.466392 18609 catalog_manager.cc:5495] T 5cfe9fdf0db749649d0651b47ccbd98c P 5e5a0e809d9842da90487a529361bc7d reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e5a0e809d9842da90487a529361bc7d (127.16.121.129). New cstate: current_term: 1 leader_uuid: "5e5a0e809d9842da90487a529361bc7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5a0e809d9842da90487a529361bc7d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 41735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 } health_report { overall_health: UNKNOWN } } }
I20260928 22:49:04.484040 19231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:04.481548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33934 (local address 127.16.121.129:41735)
0928 22:49:04.481758 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:04.481759 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:04.481767 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:49:04.481787 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:04.481787 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:04.481797 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:04.481857 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:49:04.481973 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:04.482499 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:04.483253 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:04.483464 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:04.483523 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:04.483685 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:49:04.483713 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:49:04.483755 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:04.483906 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 22:49:04.483927 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":135,"threads_started":1}
I20260928 22:49:04.489279 19231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:04.486946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:59479 (local address 127.16.121.129:41735)
0928 22:49:04.487002 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:04.487004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:04.487015 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:49:04.487126 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:04.487127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:04.487144 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:04.487183 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:49:04.487275 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:04.487743 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:04.488613 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:04.488857 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:04.488911 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:04.489058 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:49:04.489083 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:49:04.489120 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:04.489181 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 22:49:04.489216 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":13,"server-negotiator.queue_time_us":39}
I20260928 22:49:04.489351 19234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:04.486770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:41735 (local address 127.16.121.129:59479)
0928 22:49:04.486957 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:04.486970 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:04.487037 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:04.487190 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:04.487190 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:04.487192 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 22:49:04.487235 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:04.487236 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:04.487857 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:04.487859 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:04.488525 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:04.488528 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:04.488935 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:04.488936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:04.488984 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:04.488989 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 22:49:04.489140 (+ 151us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 22:49:04.489142 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 22:49:04.489180 (+ 38us) client_negotiation.cc:240] Negotiation successful
0928 22:49:04.489229 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":2,"thread_start_us":96,"threads_started":1}
I20260928 22:49:04.491580 19047 raft_consensus.cc:1261] T 5cfe9fdf0db749649d0651b47ccbd98c P c9a7dce56d024b8998f3a23395492f1c [term 1 FOLLOWER]: Refusing update from remote peer 5e5a0e809d9842da90487a529361bc7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:49:04.491603 18907 raft_consensus.cc:1261] T 5cfe9fdf0db749649d0651b47ccbd98c P 9d7ed45c903f4384989599e40b950c9f [term 1 FOLLOWER]: Refusing update from remote peer 5e5a0e809d9842da90487a529361bc7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:49:04.491815 19183 consensus_queue.cc:1035] T 5cfe9fdf0db749649d0651b47ccbd98c P 5e5a0e809d9842da90487a529361bc7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9a7dce56d024b8998f3a23395492f1c" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:49:04.491940 19139 consensus_queue.cc:1035] T 5cfe9fdf0db749649d0651b47ccbd98c P 5e5a0e809d9842da90487a529361bc7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d7ed45c903f4384989599e40b950c9f" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 22:49:04.496415 19181 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 22:49:06.016580 18610 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:49:06.016691 18610 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:49:06.017678 18610 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 622784bfe8d5446abd97cdd40c6d8115: Sending DeleteTablet for 3 replicas of tablet 5cfe9fdf0db749649d0651b47ccbd98c
I20260928 22:49:06.017951 18747 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cfe9fdf0db749649d0651b47ccbd98c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:49:06 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:33930
I20260928 22:49:06.017997 19027 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cfe9fdf0db749649d0651b47ccbd98c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:49:06 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:53014
I20260928 22:49:06.018090 18887 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cfe9fdf0db749649d0651b47ccbd98c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:49:06 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:35994
I20260928 22:49:06.018191 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 18682
I20260928 22:49:06.018215 19257 tablet_replica.cc:330] T 5cfe9fdf0db749649d0651b47ccbd98c P 5e5a0e809d9842da90487a529361bc7d: stopping tablet replica
I20260928 22:49:06.020318 19258 tablet_replica.cc:330] T 5cfe9fdf0db749649d0651b47ccbd98c P c9a7dce56d024b8998f3a23395492f1c: stopping tablet replica
I20260928 22:49:06.020407 19258 raft_consensus.cc:2229] T 5cfe9fdf0db749649d0651b47ccbd98c P c9a7dce56d024b8998f3a23395492f1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:49:06.020480 19258 raft_consensus.cc:2258] T 5cfe9fdf0db749649d0651b47ccbd98c P c9a7dce56d024b8998f3a23395492f1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:49:06.020787 19258 ts_tablet_manager.cc:1905] T 5cfe9fdf0db749649d0651b47ccbd98c P c9a7dce56d024b8998f3a23395492f1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:49:06.023234 19259 tablet_replica.cc:330] T 5cfe9fdf0db749649d0651b47ccbd98c P 9d7ed45c903f4384989599e40b950c9f: stopping tablet replica
I20260928 22:49:06.023345 19259 raft_consensus.cc:2229] T 5cfe9fdf0db749649d0651b47ccbd98c P 9d7ed45c903f4384989599e40b950c9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:49:06.023399 19259 raft_consensus.cc:2258] T 5cfe9fdf0db749649d0651b47ccbd98c P 9d7ed45c903f4384989599e40b950c9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:49:06.023409 19258 ts_tablet_manager.cc:1918] T 5cfe9fdf0db749649d0651b47ccbd98c P c9a7dce56d024b8998f3a23395492f1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 22:49:06.023484 19258 log.cc:1196] T 5cfe9fdf0db749649d0651b47ccbd98c P c9a7dce56d024b8998f3a23395492f1c: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-2/wal/wals/5cfe9fdf0db749649d0651b47ccbd98c
I20260928 22:49:06.023723 19258 ts_tablet_manager.cc:1939] T 5cfe9fdf0db749649d0651b47ccbd98c P c9a7dce56d024b8998f3a23395492f1c: Deleting consensus metadata
I20260928 22:49:06.023803 19259 ts_tablet_manager.cc:1905] T 5cfe9fdf0db749649d0651b47ccbd98c P 9d7ed45c903f4384989599e40b950c9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:49:06.024835 18597 catalog_manager.cc:4841] TS c9a7dce56d024b8998f3a23395492f1c (127.16.121.131:41721): tablet 5cfe9fdf0db749649d0651b47ccbd98c (table test-table [id=1bce01434c1c4b7dbbd744968549c2f3]) successfully deleted
I20260928 22:49:06.026513 19259 ts_tablet_manager.cc:1918] T 5cfe9fdf0db749649d0651b47ccbd98c P 9d7ed45c903f4384989599e40b950c9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 22:49:06.026628 19259 log.cc:1196] T 5cfe9fdf0db749649d0651b47ccbd98c P 9d7ed45c903f4384989599e40b950c9f: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790635733329745-16870-0/minicluster-data/ts-1/wal/wals/5cfe9fdf0db749649d0651b47ccbd98c
I20260928 22:49:06.026877 19259 ts_tablet_manager.cc:1939] T 5cfe9fdf0db749649d0651b47ccbd98c P 9d7ed45c903f4384989599e40b950c9f: Deleting consensus metadata
I20260928 22:49:06.027357 18595 catalog_manager.cc:4841] TS 9d7ed45c903f4384989599e40b950c9f (127.16.121.130:38211): tablet 5cfe9fdf0db749649d0651b47ccbd98c (table test-table [id=1bce01434c1c4b7dbbd744968549c2f3]) successfully deleted
W20260928 22:49:06.027806 18596 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.121.129:41735 (error 108)
I20260928 22:49:06.028061 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 18822
I20260928 22:49:06.035027 19261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:06.034616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:41735 (local address 127.0.0.1:33948)
0928 22:49:06.034831 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:06.034965 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.129:41735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":113,"threads_started":1}
I20260928 22:49:06.034994 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 18962
W20260928 22:49:06.035786 18596 catalog_manager.cc:4584] TS 5e5a0e809d9842da90487a529361bc7d (127.16.121.129:41735): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5cfe9fdf0db749649d0651b47ccbd98c: Network error: Client connection negotiation failed: client connection to 127.16.121.129:41735: connect: Connection refused (error 111)
I20260928 22:49:06.041170 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 18578
2026-09-28T22:49:06Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5775 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:49:06 dist-test-slave-56vw krb5kdc[19264](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:49:06 dist-test-slave-56vw krb5kdc[19264](info): set up 2 sockets
Sep 28 22:49:06 dist-test-slave-56vw krb5kdc[19264](info): commencing operation
krb5kdc: starting...
W20260928 22:49:07.103363 16870 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Sep 28 22:49:07 dist-test-slave-56vw krb5kdc[19264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:49:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:49:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.121.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
I20260928 22:49:07.243988 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:46827
--webserver_interface=127.16.121.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:36223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:46827
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:49:07.331154 19277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:07.331398 19277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:07.331423 19277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:07.332727 19277 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:49:07.332777 19277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:07.332791 19277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:07.332803 19277 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:49:07.332816 19277 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:49:07.334337 19277 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:36223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:46827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:46827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.121.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:07.334609 19277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:07.334872 19277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:07.340374 19283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:07.340381 19282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:07.340381 19285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:07.340801 19277 server_base.cc:1034] running on GCE node
I20260928 22:49:07.341063 19277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:07.341368 19277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:07.342573 19277 hybrid_clock.cc:648] HybridClock initialized: now 1790635747342554 us; error 44 us; skew 500 ppm
Sep 28 22:49:07 dist-test-slave-56vw krb5kdc[19264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635747, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:07.345634 19277 init.cc:377] Logged in from keytab as kudu/127.16.121.190@KRBTEST.COM (short username kudu)
I20260928 22:49:07.346226 19277 webserver.cc:460] Webserver started at http://127.16.121.190:33299/ using document root <none> and password file <none>
I20260928 22:49:07.346487 19277 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:07.346562 19277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:07.346737 19277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:07.348022 19277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "cb65eb499eb543279b164b93a834a5af"
format_stamp: "Formatted at 2026-09-28 22:49:07 on dist-test-slave-56vw"
server_key: "cb14e262d8a2fa292b22286008ae1ed2"
server_key_iv: "06f0e84f706799125836b394b5d1b730"
server_key_version: "encryptionkey@0"
I20260928 22:49:07.348466 19277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "cb65eb499eb543279b164b93a834a5af"
format_stamp: "Formatted at 2026-09-28 22:49:07 on dist-test-slave-56vw"
server_key: "cb14e262d8a2fa292b22286008ae1ed2"
server_key_iv: "06f0e84f706799125836b394b5d1b730"
server_key_version: "encryptionkey@0"
I20260928 22:49:07.350095 19277 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 22:49:07.351143 19292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:07.351490 19277 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:07.351594 19277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "cb65eb499eb543279b164b93a834a5af"
format_stamp: "Formatted at 2026-09-28 22:49:07 on dist-test-slave-56vw"
server_key: "cb14e262d8a2fa292b22286008ae1ed2"
server_key_iv: "06f0e84f706799125836b394b5d1b730"
server_key_version: "encryptionkey@0"
I20260928 22:49:07.351698 19277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:07.366006 19277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:07.368956 19277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:07.369087 19277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:07.373507 19277 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.190:46827
I20260928 22:49:07.373628 19344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.190:46827 every 8 connection(s)
I20260928 22:49:07.373952 19277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:49:07.374846 19345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:07.377254 19345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb65eb499eb543279b164b93a834a5af: Bootstrap starting.
I20260928 22:49:07.377883 19345 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb65eb499eb543279b164b93a834a5af: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:07.378175 19345 log.cc:824] T 00000000000000000000000000000000 P cb65eb499eb543279b164b93a834a5af: Log is configured to *not* fsync() on all Append() calls
I20260928 22:49:07.378955 19345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb65eb499eb543279b164b93a834a5af: No bootstrap required, opened a new log
I20260928 22:49:07.379799 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 19277
I20260928 22:49:07.379923 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:49:07.380136 16870 external_mini_cluster.cc:1428] Setting key e13ec848f288d0030108024a228434f8
I20260928 22:49:07.381251 19345 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb65eb499eb543279b164b93a834a5af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb65eb499eb543279b164b93a834a5af" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 46827 } }
I20260928 22:49:07.381448 19345 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb65eb499eb543279b164b93a834a5af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:07.381495 19345 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb65eb499eb543279b164b93a834a5af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb65eb499eb543279b164b93a834a5af, State: Initialized, Role: FOLLOWER
I20260928 22:49:07.381659 19345 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb65eb499eb543279b164b93a834a5af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb65eb499eb543279b164b93a834a5af" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 46827 } }
I20260928 22:49:07.381731 19345 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb65eb499eb543279b164b93a834a5af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:49:07.381755 19345 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb65eb499eb543279b164b93a834a5af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:49:07.381824 19345 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb65eb499eb543279b164b93a834a5af [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:07.382786 19345 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb65eb499eb543279b164b93a834a5af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb65eb499eb543279b164b93a834a5af" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 46827 } }
I20260928 22:49:07.382885 19345 leader_election.cc:302] T 00000000000000000000000000000000 P cb65eb499eb543279b164b93a834a5af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb65eb499eb543279b164b93a834a5af; no voters:
I20260928 22:49:07.383116 19345 leader_election.cc:288] T 00000000000000000000000000000000 P cb65eb499eb543279b164b93a834a5af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:49:07.383140 19350 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb65eb499eb543279b164b93a834a5af [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:49:07.383249 19350 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb65eb499eb543279b164b93a834a5af [term 1 LEADER]: Becoming Leader. State: Replica: cb65eb499eb543279b164b93a834a5af, State: Running, Role: LEADER
I20260928 22:49:07.383334 19350 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb65eb499eb543279b164b93a834a5af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb65eb499eb543279b164b93a834a5af" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 46827 } }
I20260928 22:49:07.383401 19345 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb65eb499eb543279b164b93a834a5af [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:49:07.383695 19351 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb65eb499eb543279b164b93a834a5af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb65eb499eb543279b164b93a834a5af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb65eb499eb543279b164b93a834a5af" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 46827 } } }
I20260928 22:49:07.383792 19351 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb65eb499eb543279b164b93a834a5af [sys.catalog]: This master's current role is: LEADER
I20260928 22:49:07.383951 19352 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb65eb499eb543279b164b93a834a5af [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb65eb499eb543279b164b93a834a5af. Latest consensus state: current_term: 1 leader_uuid: "cb65eb499eb543279b164b93a834a5af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb65eb499eb543279b164b93a834a5af" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 46827 } } }
I20260928 22:49:07.384023 19352 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb65eb499eb543279b164b93a834a5af [sys.catalog]: This master's current role is: LEADER
I20260928 22:49:07.384142 19355 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:49:07.384661 19355 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 22:49:07 dist-test-slave-56vw krb5kdc[19264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:49:07.389281 19355 catalog_manager.cc:1347] Generated new cluster ID: a4596f35062845cbada85515dbe35070
I20260928 22:49:07.389362 19355 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:49:07.402266 19349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:07.380775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54686 (local address 127.16.121.190:46827)
0928 22:49:07.380989 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:07.381000 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:07.381042 (+ 42us) server_negotiation.cc:407] Connection header received
0928 22:49:07.381203 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:07.381209 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:07.381258 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:07.381409 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:07.382217 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:07.383476 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:07.384618 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:07.385285 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:07.385717 (+ 432us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:07.388114 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:07.388120 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:07.388120 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:07.388134 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:07.400306 (+ 12172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:07.400840 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:07.400845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:07.400845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:07.400984 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:07.401211 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:07.401212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:07.401212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:07.401576 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:07.401772 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:07.401848 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 22:49:07.402002 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":90,"threads_started":1}
I20260928 22:49:07.405006 19355 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:49:07.405747 19355 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:49:07.410861 19355 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb65eb499eb543279b164b93a834a5af: Generated new TSK 0
I20260928 22:49:07.411135 19355 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.121.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab.
Entry for principal kudu/127.16.121.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab.
I20260928 22:49:07.443516 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.129:0
--local_ip_for_outbound_sockets=127.16.121.129
--webserver_interface=127.16.121.129
--webserver_port=0
--tserver_master_addrs=127.16.121.190:46827
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab
--principal=kudu/127.16.121.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:36223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:49:07.532701 19371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:07.532958 19371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:07.533001 19371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:07.534308 19371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:07.534384 19371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:07.534430 19371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.129
I20260928 22:49:07.536044 19371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:36223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab
--principal=kudu/127.16.121.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.121.129
--webserver_port=0
--tserver_master_addrs=127.16.121.190:46827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.129
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:07.536339 19371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:07.536587 19371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:07.542194 19377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:07.542241 19379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:07.542204 19376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:07.542338 19371 server_base.cc:1034] running on GCE node
I20260928 22:49:07.542778 19371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:07.543087 19371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:07.544255 19371 hybrid_clock.cc:648] HybridClock initialized: now 1790635747544246 us; error 30 us; skew 500 ppm
Sep 28 22:49:07 dist-test-slave-56vw krb5kdc[19264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635747, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:07.547026 19371 init.cc:377] Logged in from keytab as kudu/127.16.121.129@KRBTEST.COM (short username kudu)
I20260928 22:49:07.547564 19371 webserver.cc:460] Webserver started at http://127.16.121.129:37719/ using document root <none> and password file <none>
I20260928 22:49:07.547820 19371 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:07.547865 19371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:07.548008 19371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:07.549137 19371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-0/data/instance:
uuid: "47df8cfe4c444638bd2dcc0018e2e067"
format_stamp: "Formatted at 2026-09-28 22:49:07 on dist-test-slave-56vw"
server_key: "127af921e3caeeeeaeff04f88e618a92"
server_key_iv: "15852209d26d38aab23d83e6e3142cd1"
server_key_version: "encryptionkey@0"
I20260928 22:49:07.549510 19371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance:
uuid: "47df8cfe4c444638bd2dcc0018e2e067"
format_stamp: "Formatted at 2026-09-28 22:49:07 on dist-test-slave-56vw"
server_key: "127af921e3caeeeeaeff04f88e618a92"
server_key_iv: "15852209d26d38aab23d83e6e3142cd1"
server_key_version: "encryptionkey@0"
I20260928 22:49:07.550904 19371 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:49:07.551782 19386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:07.552085 19371 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:07.552165 19371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-0/wal
uuid: "47df8cfe4c444638bd2dcc0018e2e067"
format_stamp: "Formatted at 2026-09-28 22:49:07 on dist-test-slave-56vw"
server_key: "127af921e3caeeeeaeff04f88e618a92"
server_key_iv: "15852209d26d38aab23d83e6e3142cd1"
server_key_version: "encryptionkey@0"
I20260928 22:49:07.552258 19371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:07.572196 19371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:07.574764 19371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:07.574940 19371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:07.575230 19371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:49:07.575567 19371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:49:07.575623 19371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:07.575659 19371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:49:07.575686 19371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:07.581756 19371 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.129:45873
I20260928 22:49:07.581792 19499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.129:45873 every 8 connection(s)
I20260928 22:49:07.582172 19371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
Sep 28 22:49:07 dist-test-slave-56vw krb5kdc[19264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635747, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:49:07.589272 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 19371
I20260928 22:49:07.589392 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance
I20260928 22:49:07.589673 16870 external_mini_cluster.cc:1428] Setting key 3850d30bc9e0c4c484d52ed2a44ba0b8
I20260928 22:49:07.593915 19349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:07.583022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:55631 (local address 127.16.121.190:46827)
0928 22:49:07.583122 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:07.583124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:07.583186 (+ 62us) server_negotiation.cc:407] Connection header received
0928 22:49:07.583913 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:07.583914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:07.583931 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:07.583992 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:07.585189 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:07.585694 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:07.586748 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:07.586970 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:07.587101 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:07.589913 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:07.589914 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:07.589915 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:07.589926 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:07.591902 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:07.592531 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:07.592532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:07.592533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:07.592618 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:07.593013 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:07.593014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:07.593015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:07.593205 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:07.593291 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:07.593639 (+ 348us) server_negotiation.cc:299] Negotiation successful
0928 22:49:07.593754 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 22:49:07.593964 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:07.582865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:46827 (local address 127.16.121.129:55631)
0928 22:49:07.583029 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:07.583099 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:07.583780 (+ 681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:07.584094 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:07.584099 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:07.584796 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:07.585084 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:07.585090 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:07.585797 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:07.585798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:07.586638 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:07.586640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:07.587108 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:07.587109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:07.587326 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:07.588172 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:07.588181 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:07.589787 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:07.592013 (+ 2226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:07.592027 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:07.592030 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:07.592431 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:07.592730 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:07.592731 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:07.592731 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:07.592886 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:07.593287 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:07.593294 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:07.593479 (+ 185us) client_negotiation.cc:769] Sending connection context
0928 22:49:07.593579 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 22:49:07.593726 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
I20260928 22:49:07.594574 19500 heartbeater.cc:347] Connected to a master server at 127.16.121.190:46827
I20260928 22:49:07.594673 19500 heartbeater.cc:464] Registering TS with master...
I20260928 22:49:07.594884 19500 heartbeater.cc:511] Master 127.16.121.190:46827 requested a full tablet report, sending...
I20260928 22:49:07.595515 19309 ts_manager.cc:194] Registered new tserver with Master: 47df8cfe4c444638bd2dcc0018e2e067 (127.16.121.129:45873)
I20260928 22:49:07.596144 19309 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.129@KRBTEST.COM'} at 127.16.121.129:55631
WARNING: no policy specified for kudu/127.16.121.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab.
Entry for principal kudu/127.16.121.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab.
I20260928 22:49:07.620009 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.130:0
--local_ip_for_outbound_sockets=127.16.121.130
--webserver_interface=127.16.121.130
--webserver_port=0
--tserver_master_addrs=127.16.121.190:46827
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab
--principal=kudu/127.16.121.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:36223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:49:07.703419 19505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:07.703694 19505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:07.703732 19505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:07.704945 19505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:07.705016 19505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:07.705060 19505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.130
I20260928 22:49:07.706535 19505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:36223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab
--principal=kudu/127.16.121.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.121.130
--webserver_port=0
--tserver_master_addrs=127.16.121.190:46827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.130
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:07.706804 19505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:07.707072 19505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:07.712440 19511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:07.712492 19505 server_base.cc:1034] running on GCE node
W20260928 22:49:07.712445 19513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:07.712468 19510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:07.713043 19505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:07.713285 19505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:07.714432 19505 hybrid_clock.cc:648] HybridClock initialized: now 1790635747714422 us; error 31 us; skew 500 ppm
Sep 28 22:49:07 dist-test-slave-56vw krb5kdc[19264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635747, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:07.717221 19505 init.cc:377] Logged in from keytab as kudu/127.16.121.130@KRBTEST.COM (short username kudu)
I20260928 22:49:07.717684 19505 webserver.cc:460] Webserver started at http://127.16.121.130:37101/ using document root <none> and password file <none>
I20260928 22:49:07.717939 19505 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:07.718010 19505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:07.718165 19505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:07.719348 19505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-1/data/instance:
uuid: "eafdc4d19f2b40909b71c2bca8e1f74a"
format_stamp: "Formatted at 2026-09-28 22:49:07 on dist-test-slave-56vw"
server_key: "ac85489d09fa0adbe4a74ff1537c5c68"
server_key_iv: "fab1145e0a8cf166ce7ea9e29deb97f9"
server_key_version: "encryptionkey@0"
I20260928 22:49:07.719774 19505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance:
uuid: "eafdc4d19f2b40909b71c2bca8e1f74a"
format_stamp: "Formatted at 2026-09-28 22:49:07 on dist-test-slave-56vw"
server_key: "ac85489d09fa0adbe4a74ff1537c5c68"
server_key_iv: "fab1145e0a8cf166ce7ea9e29deb97f9"
server_key_version: "encryptionkey@0"
I20260928 22:49:07.721233 19505 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:49:07.722061 19520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:07.722237 19505 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:07.722347 19505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-1/wal
uuid: "eafdc4d19f2b40909b71c2bca8e1f74a"
format_stamp: "Formatted at 2026-09-28 22:49:07 on dist-test-slave-56vw"
server_key: "ac85489d09fa0adbe4a74ff1537c5c68"
server_key_iv: "fab1145e0a8cf166ce7ea9e29deb97f9"
server_key_version: "encryptionkey@0"
I20260928 22:49:07.722431 19505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:07.748694 19505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:07.751271 19505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:07.751444 19505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:07.751704 19505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:49:07.752053 19505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:49:07.752108 19505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:07.752149 19505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:49:07.752178 19505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:07.758502 19505 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.130:41959
I20260928 22:49:07.758558 19633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.130:41959 every 8 connection(s)
I20260928 22:49:07.758894 19505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
Sep 28 22:49:07 dist-test-slave-56vw krb5kdc[19264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635747, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.130@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:49:07.765918 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 19505
I20260928 22:49:07.766058 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance
I20260928 22:49:07.766302 16870 external_mini_cluster.cc:1428] Setting key 86af62b723d020f1ce8d65db79567642
I20260928 22:49:07.772569 19349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:07.759764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:54291 (local address 127.16.121.190:46827)
0928 22:49:07.759885 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:07.759887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:07.760078 (+ 191us) server_negotiation.cc:407] Connection header received
0928 22:49:07.760652 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:07.760654 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:07.760666 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:07.760723 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:07.761707 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:07.762222 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:07.763336 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:07.763565 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:07.763652 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:07.766313 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:07.766315 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:07.766316 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:07.766325 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:07.769985 (+ 3660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:07.771083 (+ 1098us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:07.771085 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:07.771085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:07.771162 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:07.771571 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:07.771572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:07.771572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:07.771742 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:07.771814 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:07.772277 (+ 463us) server_negotiation.cc:299] Negotiation successful
0928 22:49:07.772403 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 22:49:07.773245 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:07.759742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:46827 (local address 127.16.121.130:54291)
0928 22:49:07.759915 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:07.759961 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:07.760510 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:07.760807 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:07.760810 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:07.761404 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:07.761603 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:07.761607 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:07.762340 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:07.762341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:07.763250 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:07.763252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:07.763665 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:07.763666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:07.763796 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:07.764446 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:07.764455 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:07.766176 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:07.770095 (+ 3919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:07.770136 (+ 41us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:07.770140 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:07.770647 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:07.771259 (+ 612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:07.771260 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:07.771261 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:07.771474 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:07.771828 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:07.771836 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:07.772061 (+ 225us) client_negotiation.cc:769] Sending connection context
0928 22:49:07.772699 (+ 638us) client_negotiation.cc:240] Negotiation successful
0928 22:49:07.772946 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":95,"threads_started":1}
I20260928 22:49:07.773837 19634 heartbeater.cc:347] Connected to a master server at 127.16.121.190:46827
I20260928 22:49:07.773954 19634 heartbeater.cc:464] Registering TS with master...
I20260928 22:49:07.774190 19634 heartbeater.cc:511] Master 127.16.121.190:46827 requested a full tablet report, sending...
I20260928 22:49:07.774703 19309 ts_manager.cc:194] Registered new tserver with Master: eafdc4d19f2b40909b71c2bca8e1f74a (127.16.121.130:41959)
I20260928 22:49:07.775362 19309 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.130@KRBTEST.COM'} at 127.16.121.130:54291
WARNING: no policy specified for kudu/127.16.121.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab.
Entry for principal kudu/127.16.121.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab.
I20260928 22:49:07.800266 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.131:0
--local_ip_for_outbound_sockets=127.16.121.131
--webserver_interface=127.16.121.131
--webserver_port=0
--tserver_master_addrs=127.16.121.190:46827
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab
--principal=kudu/127.16.121.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:36223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:49:07.889750 19639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:07.890035 19639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:07.890081 19639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:07.891534 19639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:07.891607 19639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:07.891662 19639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.131
I20260928 22:49:07.893357 19639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:36223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab
--principal=kudu/127.16.121.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.121.131
--webserver_port=0
--tserver_master_addrs=127.16.121.190:46827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.131
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:07.893668 19639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:07.893923 19639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:07.899686 19645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:07.899693 19647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:07.899693 19644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:07.899750 19639 server_base.cc:1034] running on GCE node
I20260928 22:49:07.900071 19639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:07.900344 19639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:07.901517 19639 hybrid_clock.cc:648] HybridClock initialized: now 1790635747901475 us; error 56 us; skew 500 ppm
Sep 28 22:49:07 dist-test-slave-56vw krb5kdc[19264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635747, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:07.904343 19639 init.cc:377] Logged in from keytab as kudu/127.16.121.131@KRBTEST.COM (short username kudu)
I20260928 22:49:07.904871 19639 webserver.cc:460] Webserver started at http://127.16.121.131:34657/ using document root <none> and password file <none>
I20260928 22:49:07.905122 19639 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:07.905197 19639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:07.905356 19639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:07.906448 19639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-2/data/instance:
uuid: "ed2bd1227e5a44ad92cbe33720e12b2b"
format_stamp: "Formatted at 2026-09-28 22:49:07 on dist-test-slave-56vw"
server_key: "9c445047506303be686dfb8101a68454"
server_key_iv: "b6d54f15de34f3e54cabd3d982d52b73"
server_key_version: "encryptionkey@0"
I20260928 22:49:07.906816 19639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance:
uuid: "ed2bd1227e5a44ad92cbe33720e12b2b"
format_stamp: "Formatted at 2026-09-28 22:49:07 on dist-test-slave-56vw"
server_key: "9c445047506303be686dfb8101a68454"
server_key_iv: "b6d54f15de34f3e54cabd3d982d52b73"
server_key_version: "encryptionkey@0"
I20260928 22:49:07.908317 19639 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:49:07.909207 19654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:07.909452 19639 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:07.909557 19639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-2/wal
uuid: "ed2bd1227e5a44ad92cbe33720e12b2b"
format_stamp: "Formatted at 2026-09-28 22:49:07 on dist-test-slave-56vw"
server_key: "9c445047506303be686dfb8101a68454"
server_key_iv: "b6d54f15de34f3e54cabd3d982d52b73"
server_key_version: "encryptionkey@0"
I20260928 22:49:07.909646 19639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:07.933459 19639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:07.936103 19639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:07.936281 19639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:07.936537 19639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:49:07.936902 19639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:49:07.936956 19639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:07.937000 19639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:49:07.937027 19639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:07.943084 19639 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.131:39207
I20260928 22:49:07.943123 19767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.131:39207 every 8 connection(s)
I20260928 22:49:07.943461 19639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
I20260928 22:49:07.946038 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 19639
I20260928 22:49:07.946126 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance
I20260928 22:49:07.946457 16870 external_mini_cluster.cc:1428] Setting key b66e7a6d7a4929944247d1ab2b8cae7e
Sep 28 22:49:07 dist-test-slave-56vw krb5kdc[19264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635747, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.131@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:49:07.956693 19349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:07.944205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:35101 (local address 127.16.121.190:46827)
0928 22:49:07.944272 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:07.944274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:07.944450 (+ 176us) server_negotiation.cc:407] Connection header received
0928 22:49:07.944942 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:07.944942 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:07.944952 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:07.945019 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:07.948946 (+ 3927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:07.949529 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:07.950666 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:07.950871 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:07.950949 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:07.953370 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:07.953372 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:07.953372 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:07.953380 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:07.954974 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:07.955519 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:07.955521 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:07.955521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:07.955627 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:07.955971 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:07.955972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:07.955973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:07.956107 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:07.956167 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:07.956535 (+ 368us) server_negotiation.cc:299] Negotiation successful
0928 22:49:07.956613 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 22:49:07.956784 19770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:07.944161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:46827 (local address 127.16.121.131:35101)
0928 22:49:07.944318 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:07.944362 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:07.944847 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:07.945097 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:07.945100 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:07.945774 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:07.948837 (+ 3063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:07.948844 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:07.949635 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:07.949636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:07.950559 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:07.950561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:07.950985 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:07.950986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:07.951179 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:07.951771 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:07.951780 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:07.953274 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:07.955114 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:07.955122 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:07.955125 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:07.955392 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:07.955736 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:07.955737 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:07.955737 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:07.955860 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:07.956172 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:07.956177 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:07.956315 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 22:49:07.956405 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 22:49:07.956565 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
I20260928 22:49:07.957291 19768 heartbeater.cc:347] Connected to a master server at 127.16.121.190:46827
I20260928 22:49:07.957417 19768 heartbeater.cc:464] Registering TS with master...
I20260928 22:49:07.957640 19768 heartbeater.cc:511] Master 127.16.121.190:46827 requested a full tablet report, sending...
I20260928 22:49:07.958088 19308 ts_manager.cc:194] Registered new tserver with Master: ed2bd1227e5a44ad92cbe33720e12b2b (127.16.121.131:39207)
I20260928 22:49:07.958510 16870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:49:07.958613 19308 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.131@KRBTEST.COM'} at 127.16.121.131:35101
W20260928 22:49:07.970851 19349 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:07.969731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54700 (local address 127.16.121.190:46827)
0928 22:49:07.969841 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:07.969844 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:07.969968 (+ 124us) server_negotiation.cc:407] Connection header received
0928 22:49:07.970065 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:07.970066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:07.970085 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:07.970151 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:07.970766 (+ 615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54700: BlockingRecv error: recv got EOF from 127.0.0.1:54700 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 22:49:07.972131 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 19371
I20260928 22:49:07.977010 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 19505
I20260928 22:49:07.981514 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 19639
I20260928 22:49:07.986133 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 19277
2026-09-28T22:49:07Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1941 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635733329745-16870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:49:08 dist-test-slave-56vw krb5kdc[19780](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:49:08 dist-test-slave-56vw krb5kdc[19780](info): set up 2 sockets
Sep 28 22:49:08 dist-test-slave-56vw krb5kdc[19780](info): commencing operation
krb5kdc: starting...
W20260928 22:49:09.047363 16870 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Sep 28 22:49:09 dist-test-slave-56vw krb5kdc[19780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:49:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:49:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.121.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
I20260928 22:49:09.190445 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:33941
--webserver_interface=127.16.121.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:44953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:33941
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:49:09.278458 19793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:09.278748 19793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:09.278788 19793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:09.280056 19793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:49:09.280140 19793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:09.280162 19793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:09.280185 19793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:49:09.280215 19793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:49:09.281862 19793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:44953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:33941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:33941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.121.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:09.282187 19793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:09.282456 19793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:09.288252 19801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:09.288312 19799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:09.288255 19798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:09.288549 19793 server_base.cc:1034] running on GCE node
I20260928 22:49:09.288837 19793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:09.289178 19793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:09.290364 19793 hybrid_clock.cc:648] HybridClock initialized: now 1790635749290331 us; error 50 us; skew 500 ppm
Sep 28 22:49:09 dist-test-slave-56vw krb5kdc[19780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635749, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:09.297423 19793 init.cc:377] Logged in from keytab as kudu/127.16.121.190@KRBTEST.COM (short username kudu)
I20260928 22:49:09.298005 19793 webserver.cc:460] Webserver started at http://127.16.121.190:37073/ using document root <none> and password file <none>
I20260928 22:49:09.298249 19793 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:09.298324 19793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:09.298494 19793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:09.299821 19793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "b9d4b3826d694eb49fa9e5e5d514fa4d"
format_stamp: "Formatted at 2026-09-28 22:49:09 on dist-test-slave-56vw"
server_key: "ca56a6c91d2db6d895970cab16f6caa6"
server_key_iv: "bc4323340108a4d63a7598af52846c3f"
server_key_version: "encryptionkey@0"
I20260928 22:49:09.300235 19793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "b9d4b3826d694eb49fa9e5e5d514fa4d"
format_stamp: "Formatted at 2026-09-28 22:49:09 on dist-test-slave-56vw"
server_key: "ca56a6c91d2db6d895970cab16f6caa6"
server_key_iv: "bc4323340108a4d63a7598af52846c3f"
server_key_version: "encryptionkey@0"
I20260928 22:49:09.301697 19793 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:49:09.302668 19808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:09.302940 19793 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:09.303099 19793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "b9d4b3826d694eb49fa9e5e5d514fa4d"
format_stamp: "Formatted at 2026-09-28 22:49:09 on dist-test-slave-56vw"
server_key: "ca56a6c91d2db6d895970cab16f6caa6"
server_key_iv: "bc4323340108a4d63a7598af52846c3f"
server_key_version: "encryptionkey@0"
I20260928 22:49:09.303182 19793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:09.338899 19793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:09.341461 19793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:09.341629 19793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:09.345870 19793 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.190:33941
I20260928 22:49:09.346232 19860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.190:33941 every 8 connection(s)
I20260928 22:49:09.346290 19793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:49:09.346804 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 19793
I20260928 22:49:09.346920 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:49:09.347114 19861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:09.347134 16870 external_mini_cluster.cc:1428] Setting key e07c8ce337079cf2bfbd26813cdce08c
I20260928 22:49:09.350705 19861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9d4b3826d694eb49fa9e5e5d514fa4d: Bootstrap starting.
I20260928 22:49:09.351513 19861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9d4b3826d694eb49fa9e5e5d514fa4d: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:09.351816 19861 log.cc:824] T 00000000000000000000000000000000 P b9d4b3826d694eb49fa9e5e5d514fa4d: Log is configured to *not* fsync() on all Append() calls
I20260928 22:49:09.352579 19861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9d4b3826d694eb49fa9e5e5d514fa4d: No bootstrap required, opened a new log
Sep 28 22:49:09 dist-test-slave-56vw krb5kdc[19780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:49:09.354511 19861 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9d4b3826d694eb49fa9e5e5d514fa4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9d4b3826d694eb49fa9e5e5d514fa4d" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 33941 } }
I20260928 22:49:09.354679 19861 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9d4b3826d694eb49fa9e5e5d514fa4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:09.354730 19861 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9d4b3826d694eb49fa9e5e5d514fa4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d4b3826d694eb49fa9e5e5d514fa4d, State: Initialized, Role: FOLLOWER
I20260928 22:49:09.354914 19861 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9d4b3826d694eb49fa9e5e5d514fa4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9d4b3826d694eb49fa9e5e5d514fa4d" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 33941 } }
I20260928 22:49:09.355020 19861 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9d4b3826d694eb49fa9e5e5d514fa4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:49:09.355086 19861 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9d4b3826d694eb49fa9e5e5d514fa4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:49:09.355136 19861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9d4b3826d694eb49fa9e5e5d514fa4d [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:09.355681 19861 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9d4b3826d694eb49fa9e5e5d514fa4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9d4b3826d694eb49fa9e5e5d514fa4d" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 33941 } }
I20260928 22:49:09.355791 19861 leader_election.cc:302] T 00000000000000000000000000000000 P b9d4b3826d694eb49fa9e5e5d514fa4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9d4b3826d694eb49fa9e5e5d514fa4d; no voters:
I20260928 22:49:09.355997 19861 leader_election.cc:288] T 00000000000000000000000000000000 P b9d4b3826d694eb49fa9e5e5d514fa4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:49:09.356029 19866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9d4b3826d694eb49fa9e5e5d514fa4d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:49:09.356141 19866 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9d4b3826d694eb49fa9e5e5d514fa4d [term 1 LEADER]: Becoming Leader. State: Replica: b9d4b3826d694eb49fa9e5e5d514fa4d, State: Running, Role: LEADER
I20260928 22:49:09.356223 19866 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9d4b3826d694eb49fa9e5e5d514fa4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9d4b3826d694eb49fa9e5e5d514fa4d" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 33941 } }
I20260928 22:49:09.356278 19861 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9d4b3826d694eb49fa9e5e5d514fa4d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:49:09.356567 19866 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9d4b3826d694eb49fa9e5e5d514fa4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9d4b3826d694eb49fa9e5e5d514fa4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9d4b3826d694eb49fa9e5e5d514fa4d" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 33941 } } }
I20260928 22:49:09.356642 19866 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9d4b3826d694eb49fa9e5e5d514fa4d [sys.catalog]: This master's current role is: LEADER
I20260928 22:49:09.356751 19866 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9d4b3826d694eb49fa9e5e5d514fa4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9d4b3826d694eb49fa9e5e5d514fa4d. Latest consensus state: current_term: 1 leader_uuid: "b9d4b3826d694eb49fa9e5e5d514fa4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9d4b3826d694eb49fa9e5e5d514fa4d" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 33941 } } }
I20260928 22:49:09.356817 19866 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9d4b3826d694eb49fa9e5e5d514fa4d [sys.catalog]: This master's current role is: LEADER
W20260928 22:49:09.358002 19882 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b9d4b3826d694eb49fa9e5e5d514fa4d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:49:09.358073 19882 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:49:09.358146 19871 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:49:09.358345 19871 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:49:09.359019 19864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:09.347784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59290 (local address 127.16.121.190:33941)
0928 22:49:09.347960 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:09.347968 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:09.347994 (+ 26us) server_negotiation.cc:407] Connection header received
0928 22:49:09.348396 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:09.348402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:09.348449 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:09.348575 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:09.349394 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:09.350299 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:09.351453 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:09.352013 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:09.352146 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:09.354638 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:09.354642 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:09.354642 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:09.354653 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:09.356809 (+ 2156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:09.357276 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:09.357278 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:09.357279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:09.357391 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:09.357587 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:09.357587 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:09.357588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:09.358008 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:09.358595 (+ 587us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:09.358671 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 22:49:09.358809 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
I20260928 22:49:09.360061 16870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:49:09.360097 19871 catalog_manager.cc:1347] Generated new cluster ID: 574582ee28274b2e858aa6e70c52f8a6
I20260928 22:49:09.360132 19871 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:49:09.370675 19871 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:49:09.371459 19871 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:49:09.376263 19864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:09.372549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59292 (local address 127.16.121.190:33941)
0928 22:49:09.372595 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:09.372597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:09.372757 (+ 160us) server_negotiation.cc:407] Connection header received
0928 22:49:09.372839 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:09.372840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:09.372851 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:09.372909 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:09.373282 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:09.373775 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:09.374599 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:09.374807 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:09.374894 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:09.375738 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:09.375739 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:09.375740 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:49:09.375961 (+ 221us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:09.375981 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:49:09.375988 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:09.376045 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:09.376140 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 22:49:09.376168 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:49:09.376436 19871 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9d4b3826d694eb49fa9e5e5d514fa4d: Generated new TSK 0
I20260928 22:49:09.376623 19871 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:49:09.383297 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 19793
2026-09-28T22:49:09Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1394 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:49:09 dist-test-slave-56vw krb5kdc[19894](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:49:09 dist-test-slave-56vw krb5kdc[19894](info): set up 2 sockets
Sep 28 22:49:09 dist-test-slave-56vw krb5kdc[19894](info): commencing operation
krb5kdc: starting...
W20260928 22:49:10.467118 16870 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Sep 28 22:49:10 dist-test-slave-56vw krb5kdc[19894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:49:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:49:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.121.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
I20260928 22:49:10.598260 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:35737
--webserver_interface=127.16.121.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:42499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:35737
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:49:10.680783 19907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:10.681068 19907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:10.681110 19907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:10.682353 19907 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:49:10.682435 19907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:10.682458 19907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:10.682480 19907 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:49:10.682511 19907 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:49:10.684209 19907 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:42499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:35737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:35737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.121.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:10.684505 19907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:10.684751 19907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:10.690032 19915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:10.689988 19912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:10.689990 19913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:10.690237 19907 server_base.cc:1034] running on GCE node
I20260928 22:49:10.690627 19907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:10.690946 19907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:10.692121 19907 hybrid_clock.cc:648] HybridClock initialized: now 1790635750692115 us; error 65 us; skew 500 ppm
Sep 28 22:49:10 dist-test-slave-56vw krb5kdc[19894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635750, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:10.694998 19907 init.cc:377] Logged in from keytab as kudu/127.16.121.190@KRBTEST.COM (short username kudu)
I20260928 22:49:10.695536 19907 webserver.cc:460] Webserver started at http://127.16.121.190:41583/ using document root <none> and password file <none>
I20260928 22:49:10.695775 19907 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:10.695825 19907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:10.695976 19907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:10.697136 19907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "c443306bd66c4162ad55d156f5013691"
format_stamp: "Formatted at 2026-09-28 22:49:10 on dist-test-slave-56vw"
server_key: "67675fd56f4f4ba60273f82a58b684f0"
server_key_iv: "38d61382022e2e96b9a13845afe35837"
server_key_version: "encryptionkey@0"
I20260928 22:49:10.697641 19907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "c443306bd66c4162ad55d156f5013691"
format_stamp: "Formatted at 2026-09-28 22:49:10 on dist-test-slave-56vw"
server_key: "67675fd56f4f4ba60273f82a58b684f0"
server_key_iv: "38d61382022e2e96b9a13845afe35837"
server_key_version: "encryptionkey@0"
I20260928 22:49:10.699208 19907 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260928 22:49:10.700170 19922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:10.700418 19907 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:49:10.700536 19907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "c443306bd66c4162ad55d156f5013691"
format_stamp: "Formatted at 2026-09-28 22:49:10 on dist-test-slave-56vw"
server_key: "67675fd56f4f4ba60273f82a58b684f0"
server_key_iv: "38d61382022e2e96b9a13845afe35837"
server_key_version: "encryptionkey@0"
I20260928 22:49:10.700615 19907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:10.716027 19907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:10.719694 19907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:10.719914 19907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:10.724426 19907 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.190:35737
I20260928 22:49:10.724606 19974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.190:35737 every 8 connection(s)
I20260928 22:49:10.724910 19907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:49:10.726037 19975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:10.728950 19975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c443306bd66c4162ad55d156f5013691: Bootstrap starting.
I20260928 22:49:10.729974 19975 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c443306bd66c4162ad55d156f5013691: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:10.730322 19975 log.cc:824] T 00000000000000000000000000000000 P c443306bd66c4162ad55d156f5013691: Log is configured to *not* fsync() on all Append() calls
I20260928 22:49:10.731273 19975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c443306bd66c4162ad55d156f5013691: No bootstrap required, opened a new log
I20260928 22:49:10.733173 19975 raft_consensus.cc:348] T 00000000000000000000000000000000 P c443306bd66c4162ad55d156f5013691 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c443306bd66c4162ad55d156f5013691" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 35737 } }
I20260928 22:49:10.733355 19975 raft_consensus.cc:374] T 00000000000000000000000000000000 P c443306bd66c4162ad55d156f5013691 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:10.733379 19975 raft_consensus.cc:729] T 00000000000000000000000000000000 P c443306bd66c4162ad55d156f5013691 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c443306bd66c4162ad55d156f5013691, State: Initialized, Role: FOLLOWER
I20260928 22:49:10.733513 19975 consensus_queue.cc:260] T 00000000000000000000000000000000 P c443306bd66c4162ad55d156f5013691 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c443306bd66c4162ad55d156f5013691" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 35737 } }
I20260928 22:49:10.733565 19975 raft_consensus.cc:388] T 00000000000000000000000000000000 P c443306bd66c4162ad55d156f5013691 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:49:10.733588 19975 raft_consensus.cc:482] T 00000000000000000000000000000000 P c443306bd66c4162ad55d156f5013691 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:49:10.733618 19975 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c443306bd66c4162ad55d156f5013691 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:10.733891 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 19907
I20260928 22:49:10.734021 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:49:10.734198 16870 external_mini_cluster.cc:1428] Setting key 4d4d75ff4565618c2859d200729caeda
I20260928 22:49:10.734993 19975 raft_consensus.cc:504] T 00000000000000000000000000000000 P c443306bd66c4162ad55d156f5013691 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c443306bd66c4162ad55d156f5013691" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 35737 } }
I20260928 22:49:10.735152 19975 leader_election.cc:302] T 00000000000000000000000000000000 P c443306bd66c4162ad55d156f5013691 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c443306bd66c4162ad55d156f5013691; no voters:
I20260928 22:49:10.735546 19975 leader_election.cc:288] T 00000000000000000000000000000000 P c443306bd66c4162ad55d156f5013691 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:49:10.735752 19981 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c443306bd66c4162ad55d156f5013691 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:49:10.735882 19975 sys_catalog.cc:564] T 00000000000000000000000000000000 P c443306bd66c4162ad55d156f5013691 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:49:10.736069 19981 raft_consensus.cc:686] T 00000000000000000000000000000000 P c443306bd66c4162ad55d156f5013691 [term 1 LEADER]: Becoming Leader. State: Replica: c443306bd66c4162ad55d156f5013691, State: Running, Role: LEADER
I20260928 22:49:10.736186 19981 consensus_queue.cc:237] T 00000000000000000000000000000000 P c443306bd66c4162ad55d156f5013691 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c443306bd66c4162ad55d156f5013691" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 35737 } }
I20260928 22:49:10.736975 19981 sys_catalog.cc:455] T 00000000000000000000000000000000 P c443306bd66c4162ad55d156f5013691 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c443306bd66c4162ad55d156f5013691. Latest consensus state: current_term: 1 leader_uuid: "c443306bd66c4162ad55d156f5013691" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c443306bd66c4162ad55d156f5013691" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 35737 } } }
I20260928 22:49:10.736984 19982 sys_catalog.cc:455] T 00000000000000000000000000000000 P c443306bd66c4162ad55d156f5013691 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c443306bd66c4162ad55d156f5013691" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c443306bd66c4162ad55d156f5013691" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 35737 } } }
I20260928 22:49:10.737102 19982 sys_catalog.cc:458] T 00000000000000000000000000000000 P c443306bd66c4162ad55d156f5013691 [sys.catalog]: This master's current role is: LEADER
I20260928 22:49:10.737315 19981 sys_catalog.cc:458] T 00000000000000000000000000000000 P c443306bd66c4162ad55d156f5013691 [sys.catalog]: This master's current role is: LEADER
I20260928 22:49:10.737476 19990 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:49:10.738559 19990 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:49:10.741134 19990 catalog_manager.cc:1347] Generated new cluster ID: 6f467018d90a4d118db56beedc2b5019
I20260928 22:49:10.741200 19990 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 22:49:10 dist-test-slave-56vw krb5kdc[19894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:49:10.749557 19979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:10.735017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54776 (local address 127.16.121.190:35737)
0928 22:49:10.735544 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:10.735556 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:10.735603 (+ 47us) server_negotiation.cc:407] Connection header received
0928 22:49:10.735760 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:10.735769 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:10.735852 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:10.737145 (+ 1293us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:10.737278 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:10.738778 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:10.740055 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:10.740610 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:10.740710 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:10.744346 (+ 3636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:10.744351 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:10.744352 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:10.744370 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:10.746567 (+ 2197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:10.747038 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:10.747068 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:10.747069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:10.747230 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:10.747519 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:10.747520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:10.747521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:10.747903 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:10.748230 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:10.748902 (+ 672us) server_negotiation.cc:299] Negotiation successful
0928 22:49:10.749222 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":480,"threads_started":1}
I20260928 22:49:10.762240 19990 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:49:10.763163 19990 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:49:10.767897 19990 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c443306bd66c4162ad55d156f5013691: Generated new TSK 0
I20260928 22:49:10.768162 19990 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.121.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab.
Entry for principal kudu/127.16.121.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab.
I20260928 22:49:10.816807 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.129:0
--local_ip_for_outbound_sockets=127.16.121.129
--webserver_interface=127.16.121.129
--webserver_port=0
--tserver_master_addrs=127.16.121.190:35737
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab
--principal=kudu/127.16.121.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:42499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:49:10.904641 20001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:10.904879 20001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:10.904906 20001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:10.906195 20001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:10.906245 20001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:10.906272 20001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.129
I20260928 22:49:10.907907 20001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:42499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab
--principal=kudu/127.16.121.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.121.129
--webserver_port=0
--tserver_master_addrs=127.16.121.190:35737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.129
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:10.908188 20001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:10.908470 20001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:10.914084 20009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:10.914093 20007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:10.914103 20006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:10.914342 20001 server_base.cc:1034] running on GCE node
I20260928 22:49:10.914592 20001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:10.914829 20001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:10.916033 20001 hybrid_clock.cc:648] HybridClock initialized: now 1790635750915997 us; error 56 us; skew 500 ppm
Sep 28 22:49:10 dist-test-slave-56vw krb5kdc[19894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635750, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:10.918793 20001 init.cc:377] Logged in from keytab as kudu/127.16.121.129@KRBTEST.COM (short username kudu)
I20260928 22:49:10.919395 20001 webserver.cc:460] Webserver started at http://127.16.121.129:44817/ using document root <none> and password file <none>
I20260928 22:49:10.919658 20001 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:10.919732 20001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:10.919880 20001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:10.920970 20001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-0/data/instance:
uuid: "44f069fcadd14c43bc2eaab187fc54ea"
format_stamp: "Formatted at 2026-09-28 22:49:10 on dist-test-slave-56vw"
server_key: "24cedf1348ed3710a9c929894959c02c"
server_key_iv: "16080de056673e3d75009ff41d246380"
server_key_version: "encryptionkey@0"
I20260928 22:49:10.921339 20001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance:
uuid: "44f069fcadd14c43bc2eaab187fc54ea"
format_stamp: "Formatted at 2026-09-28 22:49:10 on dist-test-slave-56vw"
server_key: "24cedf1348ed3710a9c929894959c02c"
server_key_iv: "16080de056673e3d75009ff41d246380"
server_key_version: "encryptionkey@0"
I20260928 22:49:10.922804 20001 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:49:10.923743 20016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:10.924114 20001 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:10.924222 20001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-0/wal
uuid: "44f069fcadd14c43bc2eaab187fc54ea"
format_stamp: "Formatted at 2026-09-28 22:49:10 on dist-test-slave-56vw"
server_key: "24cedf1348ed3710a9c929894959c02c"
server_key_iv: "16080de056673e3d75009ff41d246380"
server_key_version: "encryptionkey@0"
I20260928 22:49:10.924305 20001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:10.933390 20001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:10.936412 20001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:10.936617 20001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:10.936913 20001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:49:10.937273 20001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:49:10.937345 20001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:10.937409 20001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:49:10.937438 20001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:10.944173 20001 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.129:44239
I20260928 22:49:10.944278 20129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.129:44239 every 8 connection(s)
I20260928 22:49:10.944639 20001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
Sep 28 22:49:10 dist-test-slave-56vw krb5kdc[19894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635750, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:49:10.952409 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 20001
I20260928 22:49:10.952620 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance
I20260928 22:49:10.952848 16870 external_mini_cluster.cc:1428] Setting key 0ee4f53962c71d3a83e303a36373ea06
I20260928 22:49:10.955509 19979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:10.945613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:37295 (local address 127.16.121.190:35737)
0928 22:49:10.945714 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:10.945716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:10.945893 (+ 177us) server_negotiation.cc:407] Connection header received
0928 22:49:10.946481 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:10.946483 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:10.946501 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:10.946571 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:10.947655 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:10.948194 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:10.949309 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:10.949526 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:10.949618 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:10.952101 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:10.952102 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:10.952102 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:10.952110 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:10.953781 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:10.954292 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:10.954293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:10.954294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:10.954359 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:10.954630 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:10.954631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:10.954632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:10.954792 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:10.954872 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:10.955266 (+ 394us) server_negotiation.cc:299] Negotiation successful
0928 22:49:10.955376 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 22:49:10.955580 20132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:10.945483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:35737 (local address 127.16.121.129:37295)
0928 22:49:10.945723 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:10.945795 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:10.946353 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:10.946657 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:10.946661 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:10.947325 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:10.947535 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:10.947541 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:10.948296 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:10.948297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:10.949206 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:10.949209 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:10.949631 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:10.949632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:10.949765 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:10.950348 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:10.950357 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:10.951889 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:10.953895 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:10.953905 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:10.953907 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:10.954204 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:10.954429 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:10.954430 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:10.954431 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:10.954552 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:10.954876 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:10.954884 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:10.955094 (+ 210us) client_negotiation.cc:769] Sending connection context
0928 22:49:10.955221 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 22:49:10.955370 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":113,"threads_started":1}
I20260928 22:49:10.956144 20130 heartbeater.cc:347] Connected to a master server at 127.16.121.190:35737
I20260928 22:49:10.956269 20130 heartbeater.cc:464] Registering TS with master...
I20260928 22:49:10.956471 20130 heartbeater.cc:511] Master 127.16.121.190:35737 requested a full tablet report, sending...
I20260928 22:49:10.957103 19934 ts_manager.cc:194] Registered new tserver with Master: 44f069fcadd14c43bc2eaab187fc54ea (127.16.121.129:44239)
I20260928 22:49:10.958128 19934 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.129@KRBTEST.COM'} at 127.16.121.129:37295
WARNING: no policy specified for kudu/127.16.121.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab.
Entry for principal kudu/127.16.121.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab.
I20260928 22:49:10.988147 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.130:0
--local_ip_for_outbound_sockets=127.16.121.130
--webserver_interface=127.16.121.130
--webserver_port=0
--tserver_master_addrs=127.16.121.190:35737
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab
--principal=kudu/127.16.121.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:42499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:49:11.073518 20135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:11.073835 20135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:11.073874 20135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:11.075070 20135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:11.075142 20135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:11.075187 20135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.130
I20260928 22:49:11.076733 20135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:42499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab
--principal=kudu/127.16.121.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.121.130
--webserver_port=0
--tserver_master_addrs=127.16.121.190:35737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.130
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:11.077009 20135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:11.077250 20135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:11.082511 20140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:11.082652 20143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:11.082479 20141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:11.082967 20135 server_base.cc:1034] running on GCE node
I20260928 22:49:11.083230 20135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:11.083505 20135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:11.084718 20135 hybrid_clock.cc:648] HybridClock initialized: now 1790635751084718 us; error 47 us; skew 500 ppm
Sep 28 22:49:11 dist-test-slave-56vw krb5kdc[19894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635751, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:11.087568 20135 init.cc:377] Logged in from keytab as kudu/127.16.121.130@KRBTEST.COM (short username kudu)
I20260928 22:49:11.087993 20135 webserver.cc:460] Webserver started at http://127.16.121.130:39355/ using document root <none> and password file <none>
I20260928 22:49:11.088198 20135 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:11.088239 20135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:11.088331 20135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:11.089407 20135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-1/data/instance:
uuid: "894dcce307ea4ccf96ee03ffb5c728e9"
format_stamp: "Formatted at 2026-09-28 22:49:11 on dist-test-slave-56vw"
server_key: "8283718303ea9584701f7083c3972c23"
server_key_iv: "10fd41935b1aa189a4ca1ca36a68da62"
server_key_version: "encryptionkey@0"
I20260928 22:49:11.089733 20135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance:
uuid: "894dcce307ea4ccf96ee03ffb5c728e9"
format_stamp: "Formatted at 2026-09-28 22:49:11 on dist-test-slave-56vw"
server_key: "8283718303ea9584701f7083c3972c23"
server_key_iv: "10fd41935b1aa189a4ca1ca36a68da62"
server_key_version: "encryptionkey@0"
I20260928 22:49:11.091140 20135 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 22:49:11.091998 20150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:11.092257 20135 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:11.092344 20135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-1/wal
uuid: "894dcce307ea4ccf96ee03ffb5c728e9"
format_stamp: "Formatted at 2026-09-28 22:49:11 on dist-test-slave-56vw"
server_key: "8283718303ea9584701f7083c3972c23"
server_key_iv: "10fd41935b1aa189a4ca1ca36a68da62"
server_key_version: "encryptionkey@0"
I20260928 22:49:11.092459 20135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:11.112560 20135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:11.115360 20135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:11.115545 20135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:11.115830 20135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:49:11.116204 20135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:49:11.116266 20135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:11.116323 20135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:49:11.116353 20135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:11.123027 20135 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.130:46367
I20260928 22:49:11.123142 20263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.130:46367 every 8 connection(s)
I20260928 22:49:11.123591 20135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
I20260928 22:49:11.123740 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 20135
I20260928 22:49:11.123890 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance
I20260928 22:49:11.124140 16870 external_mini_cluster.cc:1428] Setting key a8a95ba929c0bfae5a355aa9e9bd0609
Sep 28 22:49:11 dist-test-slave-56vw krb5kdc[19894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635751, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.130@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.121.131@KRBTEST.COM; defaulting to no policy
I20260928 22:49:11.134989 19979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:11.124622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:56319 (local address 127.16.121.190:35737)
0928 22:49:11.124672 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:11.124675 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:11.125033 (+ 358us) server_negotiation.cc:407] Connection header received
0928 22:49:11.125609 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:11.125610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:11.125623 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:11.125687 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:11.126714 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:11.127319 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:11.128539 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:11.128831 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:11.128942 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:11.131743 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:11.131745 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:11.131745 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:11.131755 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:11.133384 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:11.133832 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:11.133833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:11.133834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:11.133903 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:11.134195 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:11.134196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:11.134197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:11.134327 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:11.134416 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:11.134783 (+ 367us) server_negotiation.cc:299] Negotiation successful
0928 22:49:11.134889 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 22:49:11.135221 20266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:11.124609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:35737 (local address 127.16.121.130:56319)
0928 22:49:11.124867 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:11.124923 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:11.125495 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:11.125777 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:11.125784 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:11.126397 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:11.126604 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:11.126609 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:11.127430 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:11.127431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:11.128439 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:11.128442 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:11.128957 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:11.128958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:11.129095 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:11.129717 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:11.129725 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:11.131616 (+ 1891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:11.133477 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:11.133488 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:11.133491 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:11.133746 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:11.133982 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:11.133983 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:11.133983 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:11.134121 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:11.134437 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:11.134443 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:11.134632 (+ 189us) client_negotiation.cc:769] Sending connection context
0928 22:49:11.134751 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 22:49:11.134942 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":107,"threads_started":1}
I20260928 22:49:11.135870 20264 heartbeater.cc:347] Connected to a master server at 127.16.121.190:35737
I20260928 22:49:11.135977 20264 heartbeater.cc:464] Registering TS with master...
I20260928 22:49:11.136147 20264 heartbeater.cc:511] Master 127.16.121.190:35737 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.131@KRBTEST.COM" created.
I20260928 22:49:11.136693 19934 ts_manager.cc:194] Registered new tserver with Master: 894dcce307ea4ccf96ee03ffb5c728e9 (127.16.121.130:46367)
I20260928 22:49:11.137341 19934 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.130@KRBTEST.COM'} at 127.16.121.130:56319
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab.
Entry for principal kudu/127.16.121.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab.
I20260928 22:49:11.155978 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.131:0
--local_ip_for_outbound_sockets=127.16.121.131
--webserver_interface=127.16.121.131
--webserver_port=0
--tserver_master_addrs=127.16.121.190:35737
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab
--principal=kudu/127.16.121.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:42499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:49:11.244468 20269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:11.244695 20269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:11.244721 20269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:11.245875 20269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:11.245916 20269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:11.245944 20269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.131
I20260928 22:49:11.247454 20269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:42499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab
--principal=kudu/127.16.121.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.121.131
--webserver_port=0
--tserver_master_addrs=127.16.121.190:35737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.131
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:11.247653 20269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:11.247835 20269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 22:49:11.253341 20269 server_base.cc:1034] running on GCE node
W20260928 22:49:11.253324 20274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:11.253324 20275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:11.253317 20277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:11.253738 20269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:11.254037 20269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:11.255203 20269 hybrid_clock.cc:648] HybridClock initialized: now 1790635751255194 us; error 46 us; skew 500 ppm
Sep 28 22:49:11 dist-test-slave-56vw krb5kdc[19894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635751, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:11.257831 20269 init.cc:377] Logged in from keytab as kudu/127.16.121.131@KRBTEST.COM (short username kudu)
I20260928 22:49:11.258317 20269 webserver.cc:460] Webserver started at http://127.16.121.131:38735/ using document root <none> and password file <none>
I20260928 22:49:11.258598 20269 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:11.258673 20269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:11.258828 20269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:11.259985 20269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-2/data/instance:
uuid: "68a6104cac6c43639a28074e41c08c0b"
format_stamp: "Formatted at 2026-09-28 22:49:11 on dist-test-slave-56vw"
server_key: "8a94da65e2bddb4cc05f8b8e70eb8314"
server_key_iv: "56e14daac710ef3e736f112ef186f524"
server_key_version: "encryptionkey@0"
I20260928 22:49:11.260361 20269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance:
uuid: "68a6104cac6c43639a28074e41c08c0b"
format_stamp: "Formatted at 2026-09-28 22:49:11 on dist-test-slave-56vw"
server_key: "8a94da65e2bddb4cc05f8b8e70eb8314"
server_key_iv: "56e14daac710ef3e736f112ef186f524"
server_key_version: "encryptionkey@0"
I20260928 22:49:11.261816 20269 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:49:11.262735 20284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:11.262976 20269 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:11.263101 20269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-2/wal
uuid: "68a6104cac6c43639a28074e41c08c0b"
format_stamp: "Formatted at 2026-09-28 22:49:11 on dist-test-slave-56vw"
server_key: "8a94da65e2bddb4cc05f8b8e70eb8314"
server_key_iv: "56e14daac710ef3e736f112ef186f524"
server_key_version: "encryptionkey@0"
I20260928 22:49:11.263191 20269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:11.290135 20269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:11.292881 20269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:11.293058 20269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:11.293318 20269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:49:11.293644 20269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:49:11.293700 20269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:11.293745 20269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:49:11.293771 20269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:11.300019 20269 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.131:36487
I20260928 22:49:11.300066 20397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.131:36487 every 8 connection(s)
I20260928 22:49:11.300426 20269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
I20260928 22:49:11.301793 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 20269
I20260928 22:49:11.301908 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance
I20260928 22:49:11.302106 16870 external_mini_cluster.cc:1428] Setting key a0bef04fc897f166ea75a1a45ac1a93e
Sep 28 22:49:11 dist-test-slave-56vw krb5kdc[19894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635751, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.131@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:49:11.310683 19979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:11.301154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:53743 (local address 127.16.121.190:35737)
0928 22:49:11.301245 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:11.301248 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:11.301499 (+ 251us) server_negotiation.cc:407] Connection header received
0928 22:49:11.302327 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:11.302328 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:11.302341 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:11.302389 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:11.303805 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:11.304332 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:11.305407 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:11.305597 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:11.305654 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:11.308100 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:11.308101 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:11.308101 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:11.308108 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:11.309425 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:11.309795 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:11.309795 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:11.309796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:11.309848 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:11.310073 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:11.310073 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:11.310073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:11.310175 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:11.310238 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:11.310553 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 22:49:11.310616 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 22:49:11.310840 20400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:11.301127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:35737 (local address 127.16.121.131:53743)
0928 22:49:11.301345 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:11.301385 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:11.302182 (+ 797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:11.302571 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:11.302585 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:11.303511 (+ 926us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:11.303718 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:11.303724 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:11.304481 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:11.304482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:11.305318 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:11.305321 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:11.305675 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:11.305676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:11.305778 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:11.306347 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:11.306355 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:11.307980 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:11.309505 (+ 1525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:11.309512 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:11.309513 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:11.309728 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:11.309912 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:11.309913 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:11.309913 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:11.310017 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:11.310256 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:11.310262 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:11.310424 (+ 162us) client_negotiation.cc:769] Sending connection context
0928 22:49:11.310514 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 22:49:11.310638 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":110,"threads_started":1}
I20260928 22:49:11.311334 20398 heartbeater.cc:347] Connected to a master server at 127.16.121.190:35737
I20260928 22:49:11.311424 20398 heartbeater.cc:464] Registering TS with master...
I20260928 22:49:11.311651 20398 heartbeater.cc:511] Master 127.16.121.190:35737 requested a full tablet report, sending...
I20260928 22:49:11.312028 19934 ts_manager.cc:194] Registered new tserver with Master: 68a6104cac6c43639a28074e41c08c0b (127.16.121.131:36487)
I20260928 22:49:11.312568 19934 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.131@KRBTEST.COM'} at 127.16.121.131:53743
I20260928 22:49:11.314352 16870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 22:49:11 dist-test-slave-56vw krb5kdc[19894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635751, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 22:49:11 dist-test-slave-56vw krb5kdc[19894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635751, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:49:11.339247 19979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:11.330722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54778 (local address 127.16.121.190:35737)
0928 22:49:11.330822 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:11.330824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:11.330944 (+ 120us) server_negotiation.cc:407] Connection header received
0928 22:49:11.331080 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:11.331081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:11.331092 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:11.331170 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:11.331823 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:11.332354 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:11.333230 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:11.333424 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:11.333530 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:11.336023 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:11.336025 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:11.336025 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:11.336034 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:11.337912 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:11.338313 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:11.338315 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:11.338315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:11.338382 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:11.338586 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:11.338587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:11.338588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:11.338753 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:11.338859 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:11.339036 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 22:49:11.339139 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260928 22:49:11.339661 19934 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54778
I20260928 22:49:11.340209 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 20001
I20260928 22:49:11.345386 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 20135
I20260928 22:49:11.349840 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 20269
I20260928 22:49:11.354605 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 19907
2026-09-28T22:49:11Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1973 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:49:11 dist-test-slave-56vw krb5kdc[20410](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:49:11 dist-test-slave-56vw krb5kdc[20410](info): set up 2 sockets
Sep 28 22:49:11 dist-test-slave-56vw krb5kdc[20410](info): commencing operation
krb5kdc: starting...
W20260928 22:49:12.443122 16870 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Sep 28 22:49:12 dist-test-slave-56vw krb5kdc[20410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:49:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:49:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.121.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
I20260928 22:49:12.575959 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:41633
--webserver_interface=127.16.121.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:39429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:41633
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:49:12.659988 20423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:12.660280 20423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:12.660320 20423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:12.661571 20423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:49:12.661656 20423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:12.661681 20423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:12.661705 20423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:49:12.661734 20423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:49:12.663385 20423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:39429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:41633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:41633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.121.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:12.663689 20423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:12.663937 20423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:12.669216 20429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:12.669397 20423 server_base.cc:1034] running on GCE node
W20260928 22:49:12.669231 20431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:12.669216 20428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:12.669816 20423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:12.670115 20423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:12.671264 20423 hybrid_clock.cc:648] HybridClock initialized: now 1790635752671243 us; error 42 us; skew 500 ppm
Sep 28 22:49:12 dist-test-slave-56vw krb5kdc[20410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635752, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:12.674034 20423 init.cc:377] Logged in from keytab as kudu/127.16.121.190@KRBTEST.COM (short username kudu)
I20260928 22:49:12.674510 20423 webserver.cc:460] Webserver started at http://127.16.121.190:36471/ using document root <none> and password file <none>
I20260928 22:49:12.674762 20423 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:12.674836 20423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:12.674993 20423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:12.676221 20423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "93d9764d86734ed6bf8447e9d65d90f3"
format_stamp: "Formatted at 2026-09-28 22:49:12 on dist-test-slave-56vw"
server_key: "e47632f7c2cfa672272939471b7b7881"
server_key_iv: "956170f4f713d69c5b34c41a913e2948"
server_key_version: "encryptionkey@0"
I20260928 22:49:12.676620 20423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "93d9764d86734ed6bf8447e9d65d90f3"
format_stamp: "Formatted at 2026-09-28 22:49:12 on dist-test-slave-56vw"
server_key: "e47632f7c2cfa672272939471b7b7881"
server_key_iv: "956170f4f713d69c5b34c41a913e2948"
server_key_version: "encryptionkey@0"
I20260928 22:49:12.678040 20423 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:49:12.678872 20438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:12.679086 20423 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:12.679193 20423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "93d9764d86734ed6bf8447e9d65d90f3"
format_stamp: "Formatted at 2026-09-28 22:49:12 on dist-test-slave-56vw"
server_key: "e47632f7c2cfa672272939471b7b7881"
server_key_iv: "956170f4f713d69c5b34c41a913e2948"
server_key_version: "encryptionkey@0"
I20260928 22:49:12.679280 20423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:12.689416 20423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:12.691941 20423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:12.692123 20423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:12.696223 20423 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.190:41633
I20260928 22:49:12.696274 20490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.190:41633 every 8 connection(s)
I20260928 22:49:12.696828 20423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:49:12.697602 20491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:12.700340 20491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93d9764d86734ed6bf8447e9d65d90f3: Bootstrap starting.
I20260928 22:49:12.701092 20491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93d9764d86734ed6bf8447e9d65d90f3: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:12.701442 20491 log.cc:824] T 00000000000000000000000000000000 P 93d9764d86734ed6bf8447e9d65d90f3: Log is configured to *not* fsync() on all Append() calls
I20260928 22:49:12.701674 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 20423
I20260928 22:49:12.701800 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:49:12.701979 16870 external_mini_cluster.cc:1428] Setting key ce5c18dde8e58c580d03136d315152ab
I20260928 22:49:12.702392 20491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93d9764d86734ed6bf8447e9d65d90f3: No bootstrap required, opened a new log
I20260928 22:49:12.705509 20491 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93d9764d86734ed6bf8447e9d65d90f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d9764d86734ed6bf8447e9d65d90f3" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 41633 } }
I20260928 22:49:12.705718 20491 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93d9764d86734ed6bf8447e9d65d90f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:12.705754 20491 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93d9764d86734ed6bf8447e9d65d90f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93d9764d86734ed6bf8447e9d65d90f3, State: Initialized, Role: FOLLOWER
I20260928 22:49:12.705888 20491 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93d9764d86734ed6bf8447e9d65d90f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d9764d86734ed6bf8447e9d65d90f3" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 41633 } }
I20260928 22:49:12.705964 20491 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93d9764d86734ed6bf8447e9d65d90f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:49:12.705993 20491 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93d9764d86734ed6bf8447e9d65d90f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:49:12.706030 20491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93d9764d86734ed6bf8447e9d65d90f3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:12.706796 20491 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93d9764d86734ed6bf8447e9d65d90f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d9764d86734ed6bf8447e9d65d90f3" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 41633 } }
I20260928 22:49:12.706938 20491 leader_election.cc:302] T 00000000000000000000000000000000 P 93d9764d86734ed6bf8447e9d65d90f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93d9764d86734ed6bf8447e9d65d90f3; no voters:
I20260928 22:49:12.707183 20491 leader_election.cc:288] T 00000000000000000000000000000000 P 93d9764d86734ed6bf8447e9d65d90f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:49:12.707211 20496 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93d9764d86734ed6bf8447e9d65d90f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:49:12.707386 20496 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93d9764d86734ed6bf8447e9d65d90f3 [term 1 LEADER]: Becoming Leader. State: Replica: 93d9764d86734ed6bf8447e9d65d90f3, State: Running, Role: LEADER
I20260928 22:49:12.707551 20491 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93d9764d86734ed6bf8447e9d65d90f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:49:12.707553 20496 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93d9764d86734ed6bf8447e9d65d90f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d9764d86734ed6bf8447e9d65d90f3" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 41633 } }
I20260928 22:49:12.708074 20498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93d9764d86734ed6bf8447e9d65d90f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93d9764d86734ed6bf8447e9d65d90f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d9764d86734ed6bf8447e9d65d90f3" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 41633 } } }
I20260928 22:49:12.708252 20498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93d9764d86734ed6bf8447e9d65d90f3 [sys.catalog]: This master's current role is: LEADER
I20260928 22:49:12.708246 20497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93d9764d86734ed6bf8447e9d65d90f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93d9764d86734ed6bf8447e9d65d90f3. Latest consensus state: current_term: 1 leader_uuid: "93d9764d86734ed6bf8447e9d65d90f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93d9764d86734ed6bf8447e9d65d90f3" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 41633 } } }
I20260928 22:49:12.708626 20497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93d9764d86734ed6bf8447e9d65d90f3 [sys.catalog]: This master's current role is: LEADER
W20260928 22:49:12.709450 20511 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 93d9764d86734ed6bf8447e9d65d90f3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:49:12.709533 20511 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 28 22:49:12 dist-test-slave-56vw krb5kdc[20410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:49:12.711117 20505 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:49:12.711313 20505 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:49:12.713165 20505 catalog_manager.cc:1347] Generated new cluster ID: 5b9a6e5198854f3889851684564ba6b6
I20260928 22:49:12.713222 20505 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:49:12.715250 20495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:12.702570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43830 (local address 127.16.121.190:41633)
0928 22:49:12.702771 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:12.702781 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:12.702816 (+ 35us) server_negotiation.cc:407] Connection header received
0928 22:49:12.703835 (+ 1019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:12.703846 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:12.703924 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:12.704092 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:12.704838 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:12.706269 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:12.707228 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:12.707886 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:12.708029 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:12.711206 (+ 3177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:12.711211 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:12.711212 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:12.711226 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:12.713511 (+ 2285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:12.713917 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:12.713921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:12.713921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:12.714023 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:12.714239 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:12.714240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:12.714240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:12.714537 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:12.714745 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:12.714833 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 22:49:12.714973 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":103,"threads_started":1}
I20260928 22:49:12.736752 20505 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:49:12.737351 20505 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:49:12.741310 20505 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93d9764d86734ed6bf8447e9d65d90f3: Generated new TSK 0
I20260928 22:49:12.741587 20505 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.121.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab.
Entry for principal kudu/127.16.121.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab.
I20260928 22:49:12.778576 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.129:0
--local_ip_for_outbound_sockets=127.16.121.129
--webserver_interface=127.16.121.129
--webserver_port=0
--tserver_master_addrs=127.16.121.190:41633
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab
--principal=kudu/127.16.121.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:39429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:49:12.885648 20517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:12.885900 20517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:12.885931 20517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:12.887801 20517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:12.887878 20517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:12.887914 20517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.129
I20260928 22:49:12.890544 20517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:39429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab
--principal=kudu/127.16.121.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.121.129
--webserver_port=0
--tserver_master_addrs=127.16.121.190:41633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.129
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:12.890990 20517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:12.891321 20517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:12.896903 20522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:12.896940 20523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:12.896940 20525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:12.897059 20517 server_base.cc:1034] running on GCE node
I20260928 22:49:12.897364 20517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:12.897639 20517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:12.898828 20517 hybrid_clock.cc:648] HybridClock initialized: now 1790635752898817 us; error 41 us; skew 500 ppm
Sep 28 22:49:12 dist-test-slave-56vw krb5kdc[20410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635752, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:12.901755 20517 init.cc:377] Logged in from keytab as kudu/127.16.121.129@KRBTEST.COM (short username kudu)
I20260928 22:49:12.902302 20517 webserver.cc:460] Webserver started at http://127.16.121.129:38561/ using document root <none> and password file <none>
I20260928 22:49:12.902513 20517 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:12.902556 20517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:12.902652 20517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:12.903800 20517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-0/data/instance:
uuid: "aeea4f4fc20c4f60acb093e3a487b226"
format_stamp: "Formatted at 2026-09-28 22:49:12 on dist-test-slave-56vw"
server_key: "459fc781ece63da372ebe6189cedfe31"
server_key_iv: "2076b5b1d7fec608421533d843906033"
server_key_version: "encryptionkey@0"
I20260928 22:49:12.904137 20517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance:
uuid: "aeea4f4fc20c4f60acb093e3a487b226"
format_stamp: "Formatted at 2026-09-28 22:49:12 on dist-test-slave-56vw"
server_key: "459fc781ece63da372ebe6189cedfe31"
server_key_iv: "2076b5b1d7fec608421533d843906033"
server_key_version: "encryptionkey@0"
I20260928 22:49:12.905725 20517 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 22:49:12.906603 20532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:12.906838 20517 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:12.906944 20517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-0/wal
uuid: "aeea4f4fc20c4f60acb093e3a487b226"
format_stamp: "Formatted at 2026-09-28 22:49:12 on dist-test-slave-56vw"
server_key: "459fc781ece63da372ebe6189cedfe31"
server_key_iv: "2076b5b1d7fec608421533d843906033"
server_key_version: "encryptionkey@0"
I20260928 22:49:12.907013 20517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:12.950496 20517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:12.953066 20517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:12.953229 20517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:12.953451 20517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:49:12.953801 20517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:49:12.953840 20517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:12.953863 20517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:49:12.953876 20517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:12.959816 20517 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.129:44799
I20260928 22:49:12.959901 20645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.129:44799 every 8 connection(s)
I20260928 22:49:12.960245 20517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
I20260928 22:49:12.965713 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 20517
I20260928 22:49:12.965890 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance
I20260928 22:49:12.966118 16870 external_mini_cluster.cc:1428] Setting key 6fb5edabc6cc178958c1cc32b6c7d41b
Sep 28 22:49:12 dist-test-slave-56vw krb5kdc[20410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635752, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:49:12.971272 20495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:12.961021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:47869 (local address 127.16.121.190:41633)
0928 22:49:12.961101 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:12.961117 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:12.961399 (+ 282us) server_negotiation.cc:407] Connection header received
0928 22:49:12.962385 (+ 986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:12.962387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:12.962399 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:12.962459 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:12.963151 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:12.963673 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:12.964698 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:12.964912 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:12.965003 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:12.967546 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:12.967548 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:12.967548 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:12.967557 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:12.969257 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:12.969914 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:12.969915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:12.969915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:12.969982 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:12.970398 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:12.970399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:12.970399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:12.970533 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:12.970599 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:12.971007 (+ 408us) server_negotiation.cc:299] Negotiation successful
0928 22:49:12.971160 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 22:49:12.971486 20648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:12.961013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:41633 (local address 127.16.121.129:47869)
0928 22:49:12.961186 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:12.961225 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:12.962249 (+ 1024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:12.962543 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:12.962546 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:12.962817 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:12.963015 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:12.963019 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:12.963786 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:12.963787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:12.964603 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:12.964605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:12.965012 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:12.965013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:12.965156 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:12.965693 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:12.965706 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:12.967440 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:12.969426 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:12.969437 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:12.969440 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:12.969814 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:12.970060 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:12.970061 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:12.970061 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:12.970238 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:12.970619 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:12.970626 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:12.970820 (+ 194us) client_negotiation.cc:769] Sending connection context
0928 22:49:12.970958 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 22:49:12.971197 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":75,"threads_started":1}
I20260928 22:49:12.972242 20646 heartbeater.cc:347] Connected to a master server at 127.16.121.190:41633
I20260928 22:49:12.972379 20646 heartbeater.cc:464] Registering TS with master...
I20260928 22:49:12.972615 20646 heartbeater.cc:511] Master 127.16.121.190:41633 requested a full tablet report, sending...
I20260928 22:49:12.973173 20455 ts_manager.cc:194] Registered new tserver with Master: aeea4f4fc20c4f60acb093e3a487b226 (127.16.121.129:44799)
I20260928 22:49:12.973888 20455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.129@KRBTEST.COM'} at 127.16.121.129:47869
WARNING: no policy specified for kudu/127.16.121.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab.
Entry for principal kudu/127.16.121.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab.
I20260928 22:49:12.997871 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.130:0
--local_ip_for_outbound_sockets=127.16.121.130
--webserver_interface=127.16.121.130
--webserver_port=0
--tserver_master_addrs=127.16.121.190:41633
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab
--principal=kudu/127.16.121.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:39429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:49:13.082873 20651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:13.083268 20651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:13.083320 20651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:13.084695 20651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:13.084750 20651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:13.084780 20651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.130
I20260928 22:49:13.086462 20651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:39429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab
--principal=kudu/127.16.121.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.121.130
--webserver_port=0
--tserver_master_addrs=127.16.121.190:41633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.130
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:13.086745 20651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:13.087004 20651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:13.093127 20657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:13.093182 20651 server_base.cc:1034] running on GCE node
W20260928 22:49:13.093214 20656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:13.093127 20659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:13.093644 20651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:13.093978 20651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:13.095152 20651 hybrid_clock.cc:648] HybridClock initialized: now 1790635753095135 us; error 76 us; skew 500 ppm
Sep 28 22:49:13 dist-test-slave-56vw krb5kdc[20410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635753, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:13.098265 20651 init.cc:377] Logged in from keytab as kudu/127.16.121.130@KRBTEST.COM (short username kudu)
I20260928 22:49:13.098855 20651 webserver.cc:460] Webserver started at http://127.16.121.130:38147/ using document root <none> and password file <none>
I20260928 22:49:13.099205 20651 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:13.099294 20651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:13.099453 20651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:13.100626 20651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-1/data/instance:
uuid: "f63e32c6dfcb46f2b44671aaf791b0da"
format_stamp: "Formatted at 2026-09-28 22:49:13 on dist-test-slave-56vw"
server_key: "647df7797765af53457266198bca0563"
server_key_iv: "f4516a08f6861689bcb5d4440cc4786b"
server_key_version: "encryptionkey@0"
I20260928 22:49:13.101030 20651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance:
uuid: "f63e32c6dfcb46f2b44671aaf791b0da"
format_stamp: "Formatted at 2026-09-28 22:49:13 on dist-test-slave-56vw"
server_key: "647df7797765af53457266198bca0563"
server_key_iv: "f4516a08f6861689bcb5d4440cc4786b"
server_key_version: "encryptionkey@0"
I20260928 22:49:13.102694 20651 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:49:13.103828 20666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:13.104121 20651 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:13.104238 20651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-1/wal
uuid: "f63e32c6dfcb46f2b44671aaf791b0da"
format_stamp: "Formatted at 2026-09-28 22:49:13 on dist-test-slave-56vw"
server_key: "647df7797765af53457266198bca0563"
server_key_iv: "f4516a08f6861689bcb5d4440cc4786b"
server_key_version: "encryptionkey@0"
I20260928 22:49:13.104432 20651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:13.156047 20651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:13.159215 20651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:13.159443 20651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:13.159722 20651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:49:13.160079 20651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:49:13.160138 20651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:13.160183 20651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:49:13.160214 20651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:13.166832 20651 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.130:34803
I20260928 22:49:13.166976 20779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.130:34803 every 8 connection(s)
I20260928 22:49:13.167407 20651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
I20260928 22:49:13.174552 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 20651
I20260928 22:49:13.174764 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance
Sep 28 22:49:13 dist-test-slave-56vw krb5kdc[20410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635753, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.130@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:49:13.174984 16870 external_mini_cluster.cc:1428] Setting key 4e57dd535d4f85796f584c33a1e02f49
I20260928 22:49:13.179325 20495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:13.168359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:50221 (local address 127.16.121.190:41633)
0928 22:49:13.168510 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:13.168513 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:13.168852 (+ 339us) server_negotiation.cc:407] Connection header received
0928 22:49:13.169742 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:13.169744 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:13.169755 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:13.169833 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:13.170691 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:13.171495 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:13.172671 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:13.172865 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:13.172956 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:13.175538 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:13.175539 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:13.175540 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:13.175547 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:13.177448 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:13.178117 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:13.178118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:13.178119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:13.178179 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:13.178551 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:13.178553 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:13.178553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:13.178688 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:13.178756 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:13.179175 (+ 419us) server_negotiation.cc:299] Negotiation successful
0928 22:49:13.179255 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260928 22:49:13.179723 20782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:13.168318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:41633 (local address 127.16.121.130:50221)
0928 22:49:13.168589 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:13.168632 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:13.169625 (+ 993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:13.169956 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:13.169960 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:13.170278 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:13.170575 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:13.170581 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:13.171678 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:13.171679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:13.172567 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:13.172570 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:13.172980 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:13.172981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:13.173099 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:13.173772 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:13.173782 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:13.175438 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:13.177590 (+ 2152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:13.177601 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:13.177604 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:13.178008 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:13.178258 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:13.178259 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:13.178259 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:13.178436 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:13.178794 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:13.178801 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:13.178992 (+ 191us) client_negotiation.cc:769] Sending connection context
0928 22:49:13.179138 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 22:49:13.179341 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":161,"threads_started":1}
I20260928 22:49:13.180480 20780 heartbeater.cc:347] Connected to a master server at 127.16.121.190:41633
I20260928 22:49:13.180605 20780 heartbeater.cc:464] Registering TS with master...
I20260928 22:49:13.180830 20780 heartbeater.cc:511] Master 127.16.121.190:41633 requested a full tablet report, sending...
I20260928 22:49:13.181277 20455 ts_manager.cc:194] Registered new tserver with Master: f63e32c6dfcb46f2b44671aaf791b0da (127.16.121.130:34803)
I20260928 22:49:13.181685 20455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.130@KRBTEST.COM'} at 127.16.121.130:50221
WARNING: no policy specified for kudu/127.16.121.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab.
Entry for principal kudu/127.16.121.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab.
I20260928 22:49:13.208464 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.131:0
--local_ip_for_outbound_sockets=127.16.121.131
--webserver_interface=127.16.121.131
--webserver_port=0
--tserver_master_addrs=127.16.121.190:41633
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab
--principal=kudu/127.16.121.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:39429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:49:13.296690 20785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:13.296945 20785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:13.296998 20785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:13.298264 20785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:13.298338 20785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:13.298393 20785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.131
I20260928 22:49:13.300168 20785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:39429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab
--principal=kudu/127.16.121.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.121.131
--webserver_port=0
--tserver_master_addrs=127.16.121.190:41633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.131
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:13.300465 20785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:13.300711 20785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:13.306015 20791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:13.306142 20785 server_base.cc:1034] running on GCE node
W20260928 22:49:13.306102 20793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:13.306041 20790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:13.306473 20785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:13.306754 20785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:13.307940 20785 hybrid_clock.cc:648] HybridClock initialized: now 1790635753307923 us; error 52 us; skew 500 ppm
Sep 28 22:49:13 dist-test-slave-56vw krb5kdc[20410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635753, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:13.310815 20785 init.cc:377] Logged in from keytab as kudu/127.16.121.131@KRBTEST.COM (short username kudu)
I20260928 22:49:13.311337 20785 webserver.cc:460] Webserver started at http://127.16.121.131:32915/ using document root <none> and password file <none>
I20260928 22:49:13.311604 20785 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:13.311676 20785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:13.311828 20785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:13.312899 20785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-2/data/instance:
uuid: "80588bbf4af94bcd96c5fbe73e225475"
format_stamp: "Formatted at 2026-09-28 22:49:13 on dist-test-slave-56vw"
server_key: "b2defa64de8d8a45274b907b3c56381a"
server_key_iv: "c4037378e7c9aca196c6c9d530e064d3"
server_key_version: "encryptionkey@0"
I20260928 22:49:13.313274 20785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance:
uuid: "80588bbf4af94bcd96c5fbe73e225475"
format_stamp: "Formatted at 2026-09-28 22:49:13 on dist-test-slave-56vw"
server_key: "b2defa64de8d8a45274b907b3c56381a"
server_key_iv: "c4037378e7c9aca196c6c9d530e064d3"
server_key_version: "encryptionkey@0"
I20260928 22:49:13.314693 20785 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:49:13.315600 20800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:13.315778 20785 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 22:49:13.315884 20785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-2/wal
uuid: "80588bbf4af94bcd96c5fbe73e225475"
format_stamp: "Formatted at 2026-09-28 22:49:13 on dist-test-slave-56vw"
server_key: "b2defa64de8d8a45274b907b3c56381a"
server_key_iv: "c4037378e7c9aca196c6c9d530e064d3"
server_key_version: "encryptionkey@0"
I20260928 22:49:13.315984 20785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:13.330293 20785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:13.332871 20785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:13.333051 20785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:13.333308 20785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:49:13.333644 20785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:49:13.333699 20785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:13.333736 20785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:49:13.333784 20785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:13.340195 20785 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.131:35957
I20260928 22:49:13.340238 20913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.131:35957 every 8 connection(s)
I20260928 22:49:13.340579 20785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
I20260928 22:49:13.344276 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 20785
I20260928 22:49:13.344367 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance
I20260928 22:49:13.344591 16870 external_mini_cluster.cc:1428] Setting key 98f4d04ef4a7a06f0d61ba51167c1230
Sep 28 22:49:13 dist-test-slave-56vw krb5kdc[20410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635753, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.131@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:49:13.351029 20495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:13.341325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:46189 (local address 127.16.121.190:41633)
0928 22:49:13.341372 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:13.341374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:13.341617 (+ 243us) server_negotiation.cc:407] Connection header received
0928 22:49:13.342642 (+ 1025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:13.342644 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:13.342656 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:13.342722 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:13.343420 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:13.343925 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:13.345292 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:13.345626 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:13.345685 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:13.348024 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:13.348025 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:13.348025 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:13.348032 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:13.349419 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:13.349856 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:13.349857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:13.349857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:13.349907 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:13.350277 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:13.350278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:13.350279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:13.350475 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:13.350554 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:13.350889 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 22:49:13.350961 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:49:13.351315 20916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:13.341289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:41633 (local address 127.16.121.131:46189)
0928 22:49:13.341484 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:13.341529 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:13.342516 (+ 987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:13.342797 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:13.342801 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:13.343100 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:13.343295 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:13.343300 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:13.344030 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:13.344031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:13.345214 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:13.345217 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:13.345704 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:13.345705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:13.345821 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:13.346335 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:13.346344 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:13.347932 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:13.349554 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:13.349560 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:13.349562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:13.349784 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:13.350003 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:13.350003 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:13.350004 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:13.350117 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:13.350594 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:13.350600 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:13.350733 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 22:49:13.350844 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 22:49:13.350986 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":117,"threads_started":1}
I20260928 22:49:13.351971 20914 heartbeater.cc:347] Connected to a master server at 127.16.121.190:41633
I20260928 22:49:13.352098 20914 heartbeater.cc:464] Registering TS with master...
I20260928 22:49:13.352305 20914 heartbeater.cc:511] Master 127.16.121.190:41633 requested a full tablet report, sending...
I20260928 22:49:13.352702 20455 ts_manager.cc:194] Registered new tserver with Master: 80588bbf4af94bcd96c5fbe73e225475 (127.16.121.131:35957)
I20260928 22:49:13.353135 20455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.131@KRBTEST.COM'} at 127.16.121.131:46189
I20260928 22:49:13.356422 16870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 22:49:13 dist-test-slave-56vw krb5kdc[20410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 22:49:13 dist-test-slave-56vw krb5kdc[20410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.121.129@KRBTEST.COM
I20260928 22:49:13.380672 20921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:13.372065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59036 (local address 127.16.121.129:44799)
0928 22:49:13.372266 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:13.372271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:13.372282 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:49:13.372309 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:13.372312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:13.372327 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:13.372397 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:13.372992 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:13.373954 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:13.374750 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:13.375260 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:13.375322 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:13.377718 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:13.377726 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:13.377727 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:13.377734 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:13.379558 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:13.379894 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:13.379897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:13.379898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:13.379970 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:13.380151 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:13.380152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:13.380152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:13.380294 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:13.380401 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:13.380525 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 22:49:13.380588 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":104,"threads_started":1}
Sep 28 22:49:13 dist-test-slave-56vw krb5kdc[20410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:49:13.394732 20495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:13.382077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43834 (local address 127.16.121.190:41633)
0928 22:49:13.382120 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:13.382121 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:13.382153 (+ 32us) server_negotiation.cc:407] Connection header received
0928 22:49:13.382216 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:13.382217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:13.382226 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:13.382290 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:13.382905 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:13.383574 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:13.384255 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:13.384443 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:13.384513 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:13.391943 (+ 7430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:13.391946 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:13.391946 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:13.391955 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:13.393567 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:13.393923 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:13.393924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:13.393924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:13.393978 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:13.394239 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:13.394240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:13.394241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:13.394411 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:13.394508 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:13.394600 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 22:49:13.394663 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 22:49:13.395078 20455 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43834
I20260928 22:49:13.395504 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 20517
I20260928 22:49:13.401520 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 20651
I20260928 22:49:13.406289 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 20785
I20260928 22:49:13.410948 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 20423
2026-09-28T22:49:13Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2055 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:49:13 dist-test-slave-56vw krb5kdc[20926](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:49:13 dist-test-slave-56vw krb5kdc[20926](info): set up 2 sockets
Sep 28 22:49:13 dist-test-slave-56vw krb5kdc[20926](info): commencing operation
krb5kdc: starting...
W20260928 22:49:14.483232 16870 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Sep 28 22:49:14 dist-test-slave-56vw krb5kdc[20926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:49:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:49:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.121.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
I20260928 22:49:14.619896 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:40673
--webserver_interface=127.16.121.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:36097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:40673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:49:14.705585 20939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:14.705896 20939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:14.705943 20939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:14.707454 20939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:49:14.707540 20939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:14.707568 20939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:14.707592 20939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:49:14.707628 20939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:49:14.709503 20939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:36097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:40673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:40673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.121.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:14.709862 20939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:14.710147 20939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:14.715853 20944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:14.715868 20945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:14.715979 20947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:14.716216 20939 server_base.cc:1034] running on GCE node
I20260928 22:49:14.716538 20939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:14.716852 20939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:14.718055 20939 hybrid_clock.cc:648] HybridClock initialized: now 1790635754718048 us; error 40 us; skew 500 ppm
Sep 28 22:49:14 dist-test-slave-56vw krb5kdc[20926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635754, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:14.726356 20939 init.cc:377] Logged in from keytab as kudu/127.16.121.190@KRBTEST.COM (short username kudu)
I20260928 22:49:14.726763 20939 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:14.726819 20939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:14.726969 20939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:14.728188 20939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "5ffa3da6cf9d48b3bc503fa883142b0b"
format_stamp: "Formatted at 2026-09-28 22:49:14 on dist-test-slave-56vw"
server_key: "474b7dbe3e6b5533ece80fef18081140"
server_key_iv: "49f694c552c2f7fe0b8a76f09b0b38fc"
server_key_version: "encryptionkey@0"
I20260928 22:49:14.728588 20939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "5ffa3da6cf9d48b3bc503fa883142b0b"
format_stamp: "Formatted at 2026-09-28 22:49:14 on dist-test-slave-56vw"
server_key: "474b7dbe3e6b5533ece80fef18081140"
server_key_iv: "49f694c552c2f7fe0b8a76f09b0b38fc"
server_key_version: "encryptionkey@0"
I20260928 22:49:14.730039 20939 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:49:14.730976 20953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:14.731215 20939 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:14.731333 20939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "5ffa3da6cf9d48b3bc503fa883142b0b"
format_stamp: "Formatted at 2026-09-28 22:49:14 on dist-test-slave-56vw"
server_key: "474b7dbe3e6b5533ece80fef18081140"
server_key_iv: "49f694c552c2f7fe0b8a76f09b0b38fc"
server_key_version: "encryptionkey@0"
I20260928 22:49:14.731467 20939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:14.745368 20939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:14.748198 20939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:14.748365 20939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:14.752701 20939 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.190:40673
I20260928 22:49:14.752736 21005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.190:40673 every 8 connection(s)
I20260928 22:49:14.753058 20939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:49:14.754082 21006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:14.755527 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 20939
I20260928 22:49:14.755618 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:49:14.755821 16870 external_mini_cluster.cc:1428] Setting key 6d61579414417f19c6c225c532223b6a
I20260928 22:49:14.757133 21006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ffa3da6cf9d48b3bc503fa883142b0b: Bootstrap starting.
I20260928 22:49:14.757782 21006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ffa3da6cf9d48b3bc503fa883142b0b: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:14.758095 21006 log.cc:824] T 00000000000000000000000000000000 P 5ffa3da6cf9d48b3bc503fa883142b0b: Log is configured to *not* fsync() on all Append() calls
I20260928 22:49:14.758859 21006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ffa3da6cf9d48b3bc503fa883142b0b: No bootstrap required, opened a new log
I20260928 22:49:14.760805 21006 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ffa3da6cf9d48b3bc503fa883142b0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ffa3da6cf9d48b3bc503fa883142b0b" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 40673 } }
I20260928 22:49:14.761013 21006 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ffa3da6cf9d48b3bc503fa883142b0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:14.761075 21006 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ffa3da6cf9d48b3bc503fa883142b0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ffa3da6cf9d48b3bc503fa883142b0b, State: Initialized, Role: FOLLOWER
I20260928 22:49:14.761253 21006 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ffa3da6cf9d48b3bc503fa883142b0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ffa3da6cf9d48b3bc503fa883142b0b" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 40673 } }
I20260928 22:49:14.761349 21006 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ffa3da6cf9d48b3bc503fa883142b0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:49:14.761392 21006 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ffa3da6cf9d48b3bc503fa883142b0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:49:14.761444 21006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ffa3da6cf9d48b3bc503fa883142b0b [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:14.762061 21006 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ffa3da6cf9d48b3bc503fa883142b0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ffa3da6cf9d48b3bc503fa883142b0b" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 40673 } }
I20260928 22:49:14.762178 21006 leader_election.cc:302] T 00000000000000000000000000000000 P 5ffa3da6cf9d48b3bc503fa883142b0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ffa3da6cf9d48b3bc503fa883142b0b; no voters:
I20260928 22:49:14.762377 21006 leader_election.cc:288] T 00000000000000000000000000000000 P 5ffa3da6cf9d48b3bc503fa883142b0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:49:14.762449 21011 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ffa3da6cf9d48b3bc503fa883142b0b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:49:14.762619 21011 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ffa3da6cf9d48b3bc503fa883142b0b [term 1 LEADER]: Becoming Leader. State: Replica: 5ffa3da6cf9d48b3bc503fa883142b0b, State: Running, Role: LEADER
I20260928 22:49:14.762674 21006 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ffa3da6cf9d48b3bc503fa883142b0b [sys.catalog]: configured and running, proceeding with master startup.
Sep 28 22:49:14 dist-test-slave-56vw krb5kdc[20926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:49:14.762776 21011 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ffa3da6cf9d48b3bc503fa883142b0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ffa3da6cf9d48b3bc503fa883142b0b" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 40673 } }
I20260928 22:49:14.763211 21012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ffa3da6cf9d48b3bc503fa883142b0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ffa3da6cf9d48b3bc503fa883142b0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ffa3da6cf9d48b3bc503fa883142b0b" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 40673 } } }
I20260928 22:49:14.763213 21013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ffa3da6cf9d48b3bc503fa883142b0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ffa3da6cf9d48b3bc503fa883142b0b. Latest consensus state: current_term: 1 leader_uuid: "5ffa3da6cf9d48b3bc503fa883142b0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ffa3da6cf9d48b3bc503fa883142b0b" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 40673 } } }
I20260928 22:49:14.763319 21012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ffa3da6cf9d48b3bc503fa883142b0b [sys.catalog]: This master's current role is: LEADER
I20260928 22:49:14.763686 21013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ffa3da6cf9d48b3bc503fa883142b0b [sys.catalog]: This master's current role is: LEADER
I20260928 22:49:14.763849 21020 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:49:14.765234 21020 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:49:14.767138 21020 catalog_manager.cc:1347] Generated new cluster ID: d46c398f09e74354b99f1de0ba5559eb
I20260928 22:49:14.767201 21020 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:49:14.768052 21009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:14.756505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44984 (local address 127.16.121.190:40673)
0928 22:49:14.756689 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:14.756698 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:14.756726 (+ 28us) server_negotiation.cc:407] Connection header received
0928 22:49:14.757130 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:14.757137 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:14.757186 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:14.757382 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:14.758086 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:14.759016 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:14.759854 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:14.760308 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:14.760435 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:14.763412 (+ 2977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:14.763418 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:14.763419 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:14.763433 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:14.765706 (+ 2273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:14.766215 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:14.766219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:14.766219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:14.766318 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:14.766550 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:14.766552 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:14.766552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:14.766884 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:14.767125 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:14.767685 (+ 560us) server_negotiation.cc:299] Negotiation successful
0928 22:49:14.767825 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":101,"threads_started":1}
I20260928 22:49:14.776795 21020 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:49:14.777596 21020 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:49:14.789258 21020 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ffa3da6cf9d48b3bc503fa883142b0b: Generated new TSK 0
I20260928 22:49:14.789510 21020 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.121.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab.
Entry for principal kudu/127.16.121.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab.
I20260928 22:49:14.823531 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.129:0
--local_ip_for_outbound_sockets=127.16.121.129
--webserver_interface=127.16.121.129
--webserver_port=0
--tserver_master_addrs=127.16.121.190:40673
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab
--principal=kudu/127.16.121.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:36097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:49:14.912221 21032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:14.912848 21032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:14.912956 21032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:14.914688 21032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:14.914788 21032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:14.914860 21032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.129
I20260928 22:49:14.917191 21032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:36097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab
--principal=kudu/127.16.121.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.121.129
--webserver_port=0
--tserver_master_addrs=127.16.121.190:40673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.129
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:14.917711 21032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:14.918059 21032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:14.925242 21038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:14.925531 21037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:14.925295 21032 server_base.cc:1034] running on GCE node
W20260928 22:49:14.925271 21040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:14.925879 21032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:14.926261 21032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:14.927527 21032 hybrid_clock.cc:648] HybridClock initialized: now 1790635754927515 us; error 32 us; skew 500 ppm
Sep 28 22:49:14 dist-test-slave-56vw krb5kdc[20926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635754, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:14.931308 21032 init.cc:377] Logged in from keytab as kudu/127.16.121.129@KRBTEST.COM (short username kudu)
I20260928 22:49:14.931969 21032 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:14.932068 21032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:14.932250 21032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:14.933554 21032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-0/data/instance:
uuid: "94db7207ea574eb8a2796882d8fcfe29"
format_stamp: "Formatted at 2026-09-28 22:49:14 on dist-test-slave-56vw"
server_key: "d2a5616afeda33465d8de8cc7e556ecc"
server_key_iv: "3dbb87860d78293c31985c78aa724fbe"
server_key_version: "encryptionkey@0"
I20260928 22:49:14.934057 21032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance:
uuid: "94db7207ea574eb8a2796882d8fcfe29"
format_stamp: "Formatted at 2026-09-28 22:49:14 on dist-test-slave-56vw"
server_key: "d2a5616afeda33465d8de8cc7e556ecc"
server_key_iv: "3dbb87860d78293c31985c78aa724fbe"
server_key_version: "encryptionkey@0"
I20260928 22:49:14.936467 21032 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.005s
I20260928 22:49:14.937971 21046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:14.938293 21032 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:49:14.938442 21032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-0/wal
uuid: "94db7207ea574eb8a2796882d8fcfe29"
format_stamp: "Formatted at 2026-09-28 22:49:14 on dist-test-slave-56vw"
server_key: "d2a5616afeda33465d8de8cc7e556ecc"
server_key_iv: "3dbb87860d78293c31985c78aa724fbe"
server_key_version: "encryptionkey@0"
I20260928 22:49:14.938792 21032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:14.960299 21032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:14.963660 21032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:14.963856 21032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:14.964175 21032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:49:14.964586 21032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:49:14.964650 21032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:14.964695 21032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:49:14.964723 21032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:14.974888 21032 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.129:35417
I20260928 22:49:14.974936 21159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.129:35417 every 8 connection(s)
I20260928 22:49:14.975591 21032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
I20260928 22:49:14.979987 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 21032
I20260928 22:49:14.980118 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance
I20260928 22:49:14.980471 16870 external_mini_cluster.cc:1428] Setting key f88f4b40d4f0196c77a7c2e6547f44e6
Sep 28 22:49:14 dist-test-slave-56vw krb5kdc[20926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635754, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:49:14.989946 21009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:14.976973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:41817 (local address 127.16.121.190:40673)
0928 22:49:14.977032 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:14.977034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:14.977136 (+ 102us) server_negotiation.cc:407] Connection header received
0928 22:49:14.978031 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:14.978033 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:14.978046 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:14.978132 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:14.979511 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:14.980510 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:14.981873 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:14.982226 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:14.982330 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:14.986364 (+ 4034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:14.986366 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:14.986366 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:14.986379 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:14.988188 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:14.988735 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:14.988736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:14.988737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:14.988810 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:14.989130 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:14.989131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:14.989131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:14.989277 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:14.989368 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:14.989721 (+ 353us) server_negotiation.cc:299] Negotiation successful
0928 22:49:14.989829 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 22:49:14.990212 21162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:14.976803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:40673 (local address 127.16.121.129:41817)
0928 22:49:14.977004 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:14.977050 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:14.977662 (+ 612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:14.978316 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:14.978321 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:14.978656 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:14.979365 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:14.979374 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:14.980714 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:14.980715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:14.981774 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:14.981778 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:14.982372 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:14.982373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:14.982510 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:14.983355 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:14.983369 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:14.986186 (+ 2817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:14.988326 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:14.988338 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:14.988341 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:14.988638 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:14.988891 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:14.988892 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:14.988893 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:14.989045 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:14.989373 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:14.989381 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:14.989563 (+ 182us) client_negotiation.cc:769] Sending connection context
0928 22:49:14.989695 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 22:49:14.989922 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":116,"threads_started":1}
I20260928 22:49:14.991130 21160 heartbeater.cc:347] Connected to a master server at 127.16.121.190:40673
I20260928 22:49:14.991261 21160 heartbeater.cc:464] Registering TS with master...
I20260928 22:49:14.991469 21160 heartbeater.cc:511] Master 127.16.121.190:40673 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.121.130@KRBTEST.COM; defaulting to no policy
I20260928 22:49:14.992218 20970 ts_manager.cc:194] Registered new tserver with Master: 94db7207ea574eb8a2796882d8fcfe29 (127.16.121.129:35417)
I20260928 22:49:14.992887 20970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.129@KRBTEST.COM'} at 127.16.121.129:41817
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab.
Entry for principal kudu/127.16.121.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab.
I20260928 22:49:15.032784 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.130:0
--local_ip_for_outbound_sockets=127.16.121.130
--webserver_interface=127.16.121.130
--webserver_port=0
--tserver_master_addrs=127.16.121.190:40673
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab
--principal=kudu/127.16.121.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:36097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:49:15.134567 21165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:15.135306 21165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:15.135474 21165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:15.137315 21165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:15.137382 21165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:15.137419 21165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.130
I20260928 22:49:15.139890 21165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:36097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab
--principal=kudu/127.16.121.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.121.130
--webserver_port=0
--tserver_master_addrs=127.16.121.190:40673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.130
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:15.140149 21165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:15.140372 21165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:15.146764 21171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:15.146826 21173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:15.146764 21170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:15.147313 21165 server_base.cc:1034] running on GCE node
I20260928 22:49:15.147555 21165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:15.147819 21165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:15.149051 21165 hybrid_clock.cc:648] HybridClock initialized: now 1790635755149001 us; error 29 us; skew 500 ppm
Sep 28 22:49:15 dist-test-slave-56vw krb5kdc[20926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635755, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:15.152011 21165 init.cc:377] Logged in from keytab as kudu/127.16.121.130@KRBTEST.COM (short username kudu)
I20260928 22:49:15.152465 21165 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:15.152526 21165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:15.152676 21165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:15.153915 21165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-1/data/instance:
uuid: "0ddf361e77374536897979c39e10bcb2"
format_stamp: "Formatted at 2026-09-28 22:49:15 on dist-test-slave-56vw"
server_key: "a9ebbdf0e4ad82c7112d78d4b88ac81d"
server_key_iv: "2f4b5c6fa0e299b7451741535b16c8b5"
server_key_version: "encryptionkey@0"
I20260928 22:49:15.154307 21165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance:
uuid: "0ddf361e77374536897979c39e10bcb2"
format_stamp: "Formatted at 2026-09-28 22:49:15 on dist-test-slave-56vw"
server_key: "a9ebbdf0e4ad82c7112d78d4b88ac81d"
server_key_iv: "2f4b5c6fa0e299b7451741535b16c8b5"
server_key_version: "encryptionkey@0"
I20260928 22:49:15.155911 21165 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:49:15.156812 21179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:15.157083 21165 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:15.157169 21165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-1/wal
uuid: "0ddf361e77374536897979c39e10bcb2"
format_stamp: "Formatted at 2026-09-28 22:49:15 on dist-test-slave-56vw"
server_key: "a9ebbdf0e4ad82c7112d78d4b88ac81d"
server_key_iv: "2f4b5c6fa0e299b7451741535b16c8b5"
server_key_version: "encryptionkey@0"
I20260928 22:49:15.157225 21165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:15.169328 21165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:15.172587 21165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:15.172793 21165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:15.173187 21165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:49:15.173579 21165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:49:15.173643 21165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:15.173681 21165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:49:15.173712 21165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:15.181272 21165 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.130:44087
I20260928 22:49:15.181304 21292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.130:44087 every 8 connection(s)
I20260928 22:49:15.181638 21165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
Sep 28 22:49:15 dist-test-slave-56vw krb5kdc[20926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635755, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.130@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:49:15.190006 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 21165
I20260928 22:49:15.190120 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance
I20260928 22:49:15.190315 16870 external_mini_cluster.cc:1428] Setting key 83c197dace87a8ed3b0752fe92a0e237
I20260928 22:49:15.192595 21009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:15.182506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:60787 (local address 127.16.121.190:40673)
0928 22:49:15.182603 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:15.182605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:15.182763 (+ 158us) server_negotiation.cc:407] Connection header received
0928 22:49:15.183315 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:15.183317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:15.183329 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:15.183477 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:15.184645 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:15.185174 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:15.186421 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:15.186616 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:15.186680 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:15.189003 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:15.189004 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:15.189004 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:15.189012 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:15.190715 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:15.191296 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:15.191297 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:15.191297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:15.191365 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:15.191763 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:15.191764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:15.191765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:15.191923 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:15.191998 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:15.192288 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 22:49:15.192462 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 22:49:15.192646 21295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:15.182453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:40673 (local address 127.16.121.130:60787)
0928 22:49:15.182626 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:15.182671 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:15.183189 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:15.183594 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:15.183598 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:15.183893 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:15.184527 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:15.184534 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:15.185306 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:15.185308 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:15.186317 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:15.186320 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:15.186696 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:15.186697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:15.186802 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:15.187480 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:15.187489 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:15.188912 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:15.190792 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:15.190801 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:15.190803 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:15.191219 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:15.191535 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:15.191537 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:15.191537 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:15.191670 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:15.192003 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:15.192009 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:15.192144 (+ 135us) client_negotiation.cc:769] Sending connection context
0928 22:49:15.192229 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 22:49:15.192392 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":85,"threads_started":1}
I20260928 22:49:15.193207 21293 heartbeater.cc:347] Connected to a master server at 127.16.121.190:40673
I20260928 22:49:15.193331 21293 heartbeater.cc:464] Registering TS with master...
I20260928 22:49:15.193518 21293 heartbeater.cc:511] Master 127.16.121.190:40673 requested a full tablet report, sending...
I20260928 22:49:15.193917 20970 ts_manager.cc:194] Registered new tserver with Master: 0ddf361e77374536897979c39e10bcb2 (127.16.121.130:44087)
I20260928 22:49:15.194319 20970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.130@KRBTEST.COM'} at 127.16.121.130:60787
WARNING: no policy specified for kudu/127.16.121.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab.
Entry for principal kudu/127.16.121.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab.
I20260928 22:49:15.224088 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.131:0
--local_ip_for_outbound_sockets=127.16.121.131
--webserver_interface=127.16.121.131
--webserver_port=0
--tserver_master_addrs=127.16.121.190:40673
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab
--principal=kudu/127.16.121.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:36097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:49:15.311604 21298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:15.311868 21298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:15.311908 21298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:15.313112 21298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:15.313186 21298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:15.313234 21298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.131
I20260928 22:49:15.314832 21298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:36097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab
--principal=kudu/127.16.121.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.121.131
--webserver_port=0
--tserver_master_addrs=127.16.121.190:40673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.131
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:15.315138 21298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:15.315383 21298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:15.320566 21303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:15.320587 21304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:15.320569 21306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:15.320976 21298 server_base.cc:1034] running on GCE node
I20260928 22:49:15.321202 21298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:15.321404 21298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:15.322643 21298 hybrid_clock.cc:648] HybridClock initialized: now 1790635755322600 us; error 55 us; skew 500 ppm
Sep 28 22:49:15 dist-test-slave-56vw krb5kdc[20926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635755, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:15.325295 21298 init.cc:377] Logged in from keytab as kudu/127.16.121.131@KRBTEST.COM (short username kudu)
I20260928 22:49:15.325660 21298 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:15.325708 21298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:15.325809 21298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:15.326905 21298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-2/data/instance:
uuid: "564a049f0dfc4231b11f38c6d9d1453a"
format_stamp: "Formatted at 2026-09-28 22:49:15 on dist-test-slave-56vw"
server_key: "d8d433c04d951f7462de854d5329c56a"
server_key_iv: "e3b783a350e918f59deb9f56c3a4ffa1"
server_key_version: "encryptionkey@0"
I20260928 22:49:15.327442 21298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance:
uuid: "564a049f0dfc4231b11f38c6d9d1453a"
format_stamp: "Formatted at 2026-09-28 22:49:15 on dist-test-slave-56vw"
server_key: "d8d433c04d951f7462de854d5329c56a"
server_key_iv: "e3b783a350e918f59deb9f56c3a4ffa1"
server_key_version: "encryptionkey@0"
I20260928 22:49:15.328972 21298 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:15.329879 21312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:15.330090 21298 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:15.330188 21298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-2/wal
uuid: "564a049f0dfc4231b11f38c6d9d1453a"
format_stamp: "Formatted at 2026-09-28 22:49:15 on dist-test-slave-56vw"
server_key: "d8d433c04d951f7462de854d5329c56a"
server_key_iv: "e3b783a350e918f59deb9f56c3a4ffa1"
server_key_version: "encryptionkey@0"
I20260928 22:49:15.330251 21298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:15.339556 21298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:15.342933 21298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:15.343182 21298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:15.343446 21298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:49:15.343806 21298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:49:15.343839 21298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:15.343863 21298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:49:15.343878 21298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:15.349766 21298 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.131:45529
I20260928 22:49:15.349804 21425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.131:45529 every 8 connection(s)
I20260928 22:49:15.350117 21298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
Sep 28 22:49:15 dist-test-slave-56vw krb5kdc[20926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635755, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.131@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:49:15.360404 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 21298
I20260928 22:49:15.360564 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance
I20260928 22:49:15.360828 16870 external_mini_cluster.cc:1428] Setting key f2fe19ea67bf355e48f4af677903ef40
I20260928 22:49:15.361927 21009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:15.350964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:43817 (local address 127.16.121.190:40673)
0928 22:49:15.351080 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:15.351082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:15.351526 (+ 444us) server_negotiation.cc:407] Connection header received
0928 22:49:15.352057 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:15.352059 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:15.352072 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:15.352124 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:15.353273 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:15.353874 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:15.355079 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:15.355304 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:15.355412 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:15.358181 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:15.358186 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:15.358187 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:15.358195 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:15.359876 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:15.360570 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:15.360572 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:15.360572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:15.360655 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:15.361064 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:15.361066 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:15.361066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:15.361250 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:15.361365 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:15.361746 (+ 381us) server_negotiation.cc:299] Negotiation successful
0928 22:49:15.361845 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 22:49:15.362263 21428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:15.351003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:40673 (local address 127.16.121.131:43817)
0928 22:49:15.351292 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:15.351335 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:15.351925 (+ 590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:15.352216 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:15.352221 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:15.352519 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:15.353144 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:15.353150 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:15.353984 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:15.353985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:15.354943 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:15.354946 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:15.355439 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:15.355440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:15.355575 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:15.356203 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:15.356217 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:15.358049 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:15.359995 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:15.360003 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:15.360006 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:15.360426 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:15.360756 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:15.360757 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:15.360758 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:15.360942 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:15.361355 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:15.361363 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:15.361589 (+ 226us) client_negotiation.cc:769] Sending connection context
0928 22:49:15.361722 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 22:49:15.362020 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":174,"threads_started":1}
I20260928 22:49:15.362848 21426 heartbeater.cc:347] Connected to a master server at 127.16.121.190:40673
I20260928 22:49:15.362965 21426 heartbeater.cc:464] Registering TS with master...
I20260928 22:49:15.363174 21426 heartbeater.cc:511] Master 127.16.121.190:40673 requested a full tablet report, sending...
I20260928 22:49:15.363667 20970 ts_manager.cc:194] Registered new tserver with Master: 564a049f0dfc4231b11f38c6d9d1453a (127.16.121.131:45529)
I20260928 22:49:15.364079 20970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.131@KRBTEST.COM'} at 127.16.121.131:43817
I20260928 22:49:15.372943 16870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:49:15.381870 21009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:15.374418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44986 (local address 127.16.121.190:40673)
0928 22:49:15.374496 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:15.374499 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:15.374651 (+ 152us) server_negotiation.cc:407] Connection header received
0928 22:49:15.374753 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:15.374754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:15.374767 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:15.374820 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:15.375590 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:15.376145 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:15.377000 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:15.377219 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:15.377315 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:15.378731 (+ 1416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:15.378733 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:15.378733 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:15.378741 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:15.380456 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:15.380889 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:15.380890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:15.380890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:15.380961 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:15.381188 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:15.381189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:15.381189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:15.381369 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:15.381499 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:15.381657 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 22:49:15.381753 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 22:49:15.384174 20970 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44986:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:49:15.386030 20970 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:49:15.393814 21440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:15.389440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37880 (local address 127.16.121.129:35417)
0928 22:49:15.389643 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:15.389649 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:15.389831 (+ 182us) server_negotiation.cc:407] Connection header received
0928 22:49:15.390112 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:15.390116 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:15.390235 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:15.390310 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:49:15.390408 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:15.391541 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:15.392833 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:15.393389 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:15.393486 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:15.393548 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:15.393673 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 22:49:15.393711 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":86,"threads_started":1}
I20260928 22:49:15.393862 21439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:15.389336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.131:45529 (local address 127.0.0.1:44864)
0928 22:49:15.389961 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:15.389970 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:15.390008 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:15.390272 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:15.390273 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:15.390275 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:49:15.390336 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:15.390338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:15.391312 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:15.391313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:15.392568 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:15.392571 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:15.393584 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:15.393585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:15.393648 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:15.393653 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:49:15.393703 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 22:49:15.393751 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"mutex_wait_us":8,"thread_start_us":131,"threads_started":1}
I20260928 22:49:15.394114 21438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:15.389334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.130:44087 (local address 127.0.0.1:44862)
0928 22:49:15.389857 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:15.389867 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:15.389930 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:15.390175 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:15.390177 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:15.390180 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:49:15.390261 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:15.390264 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:15.391665 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:15.391666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:15.392916 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:15.392918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:15.393885 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:15.393886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:15.393947 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:15.393951 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:49:15.394011 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 22:49:15.394053 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"mutex_wait_us":52,"thread_start_us":97,"threads_started":1}
I20260928 22:49:15.394156 21441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:15.389682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44862 (local address 127.16.121.130:44087)
0928 22:49:15.389835 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:15.389840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:15.389930 (+ 90us) server_negotiation.cc:407] Connection header received
0928 22:49:15.390015 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:15.390017 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:15.390035 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:15.390113 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:49:15.390357 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:15.391565 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:15.393004 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:15.393730 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:15.393791 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:15.393841 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:15.394034 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 22:49:15.394082 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
I20260928 22:49:15.394203 21442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:15.389664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44864 (local address 127.16.121.131:45529)
0928 22:49:15.390051 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:15.390056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:15.390067 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:49:15.390102 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:15.390104 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:15.390117 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:15.390188 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:49:15.390416 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:15.391216 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:15.392662 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:15.393422 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:15.393495 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:15.393548 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:15.394060 (+ 512us) server_negotiation.cc:299] Negotiation successful
0928 22:49:15.394104 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":87,"threads_started":1}
I20260928 22:49:15.393862 21437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:15.389333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:35417 (local address 127.0.0.1:37880)
0928 22:49:15.389717 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:15.389766 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:15.389851 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:15.390318 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:15.390319 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:15.390321 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:49:15.390350 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:15.390351 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:15.391658 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:15.391659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:15.392745 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:15.392748 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:15.393485 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:15.393487 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:15.393567 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:15.393581 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 22:49:15.393653 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 22:49:15.393710 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":36,"thread_start_us":165,"threads_started":1}
I20260928 22:49:15.394837 21360 tablet_service.cc:1467] Processing CreateTablet for tablet b239182a448f42e29bee1e97147c06a1 (DEFAULT_TABLE table=test-table [id=6d1fe742d6f24e1197378a6c2b8207cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:49:15.394861 21227 tablet_service.cc:1467] Processing CreateTablet for tablet b239182a448f42e29bee1e97147c06a1 (DEFAULT_TABLE table=test-table [id=6d1fe742d6f24e1197378a6c2b8207cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:49:15.395293 21227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b239182a448f42e29bee1e97147c06a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:15.395414 21360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b239182a448f42e29bee1e97147c06a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:15.395525 21094 tablet_service.cc:1467] Processing CreateTablet for tablet b239182a448f42e29bee1e97147c06a1 (DEFAULT_TABLE table=test-table [id=6d1fe742d6f24e1197378a6c2b8207cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:49:15.395833 21094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b239182a448f42e29bee1e97147c06a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:15.398878 21443 tablet_bootstrap.cc:492] T b239182a448f42e29bee1e97147c06a1 P 564a049f0dfc4231b11f38c6d9d1453a: Bootstrap starting.
I20260928 22:49:15.399546 21445 tablet_bootstrap.cc:492] T b239182a448f42e29bee1e97147c06a1 P 94db7207ea574eb8a2796882d8fcfe29: Bootstrap starting.
I20260928 22:49:15.399698 21443 tablet_bootstrap.cc:654] T b239182a448f42e29bee1e97147c06a1 P 564a049f0dfc4231b11f38c6d9d1453a: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:15.400143 21445 tablet_bootstrap.cc:654] T b239182a448f42e29bee1e97147c06a1 P 94db7207ea574eb8a2796882d8fcfe29: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:15.400260 21443 log.cc:824] T b239182a448f42e29bee1e97147c06a1 P 564a049f0dfc4231b11f38c6d9d1453a: Log is configured to *not* fsync() on all Append() calls
I20260928 22:49:15.400672 21445 log.cc:824] T b239182a448f42e29bee1e97147c06a1 P 94db7207ea574eb8a2796882d8fcfe29: Log is configured to *not* fsync() on all Append() calls
I20260928 22:49:15.401011 21444 tablet_bootstrap.cc:492] T b239182a448f42e29bee1e97147c06a1 P 0ddf361e77374536897979c39e10bcb2: Bootstrap starting.
I20260928 22:49:15.401074 21443 tablet_bootstrap.cc:492] T b239182a448f42e29bee1e97147c06a1 P 564a049f0dfc4231b11f38c6d9d1453a: No bootstrap required, opened a new log
I20260928 22:49:15.401147 21443 ts_tablet_manager.cc:1397] T b239182a448f42e29bee1e97147c06a1 P 564a049f0dfc4231b11f38c6d9d1453a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 22:49:15.401396 21445 tablet_bootstrap.cc:492] T b239182a448f42e29bee1e97147c06a1 P 94db7207ea574eb8a2796882d8fcfe29: No bootstrap required, opened a new log
I20260928 22:49:15.401472 21445 ts_tablet_manager.cc:1397] T b239182a448f42e29bee1e97147c06a1 P 94db7207ea574eb8a2796882d8fcfe29: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:49:15.402837 21443 raft_consensus.cc:348] T b239182a448f42e29bee1e97147c06a1 P 564a049f0dfc4231b11f38c6d9d1453a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564a049f0dfc4231b11f38c6d9d1453a" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 45529 } } peers { permanent_uuid: "94db7207ea574eb8a2796882d8fcfe29" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 35417 } } peers { permanent_uuid: "0ddf361e77374536897979c39e10bcb2" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 44087 } }
I20260928 22:49:15.403009 21443 raft_consensus.cc:374] T b239182a448f42e29bee1e97147c06a1 P 564a049f0dfc4231b11f38c6d9d1453a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:15.403079 21443 raft_consensus.cc:729] T b239182a448f42e29bee1e97147c06a1 P 564a049f0dfc4231b11f38c6d9d1453a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 564a049f0dfc4231b11f38c6d9d1453a, State: Initialized, Role: FOLLOWER
I20260928 22:49:15.403211 21443 consensus_queue.cc:260] T b239182a448f42e29bee1e97147c06a1 P 564a049f0dfc4231b11f38c6d9d1453a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564a049f0dfc4231b11f38c6d9d1453a" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 45529 } } peers { permanent_uuid: "94db7207ea574eb8a2796882d8fcfe29" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 35417 } } peers { permanent_uuid: "0ddf361e77374536897979c39e10bcb2" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 44087 } }
I20260928 22:49:15.403329 21445 raft_consensus.cc:348] T b239182a448f42e29bee1e97147c06a1 P 94db7207ea574eb8a2796882d8fcfe29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564a049f0dfc4231b11f38c6d9d1453a" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 45529 } } peers { permanent_uuid: "94db7207ea574eb8a2796882d8fcfe29" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 35417 } } peers { permanent_uuid: "0ddf361e77374536897979c39e10bcb2" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 44087 } }
I20260928 22:49:15.403707 21426 heartbeater.cc:503] Master 127.16.121.190:40673 was elected leader, sending a full tablet report...
I20260928 22:49:15.403733 21443 ts_tablet_manager.cc:1428] T b239182a448f42e29bee1e97147c06a1 P 564a049f0dfc4231b11f38c6d9d1453a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 22:49:15.403664 21445 raft_consensus.cc:374] T b239182a448f42e29bee1e97147c06a1 P 94db7207ea574eb8a2796882d8fcfe29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:15.403832 21445 raft_consensus.cc:729] T b239182a448f42e29bee1e97147c06a1 P 94db7207ea574eb8a2796882d8fcfe29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94db7207ea574eb8a2796882d8fcfe29, State: Initialized, Role: FOLLOWER
I20260928 22:49:15.404043 21445 consensus_queue.cc:260] T b239182a448f42e29bee1e97147c06a1 P 94db7207ea574eb8a2796882d8fcfe29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564a049f0dfc4231b11f38c6d9d1453a" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 45529 } } peers { permanent_uuid: "94db7207ea574eb8a2796882d8fcfe29" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 35417 } } peers { permanent_uuid: "0ddf361e77374536897979c39e10bcb2" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 44087 } }
I20260928 22:49:15.404343 21445 ts_tablet_manager.cc:1428] T b239182a448f42e29bee1e97147c06a1 P 94db7207ea574eb8a2796882d8fcfe29: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:49:15.404428 21160 heartbeater.cc:503] Master 127.16.121.190:40673 was elected leader, sending a full tablet report...
I20260928 22:49:15.405313 21444 tablet_bootstrap.cc:654] T b239182a448f42e29bee1e97147c06a1 P 0ddf361e77374536897979c39e10bcb2: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:15.405949 21444 log.cc:824] T b239182a448f42e29bee1e97147c06a1 P 0ddf361e77374536897979c39e10bcb2: Log is configured to *not* fsync() on all Append() calls
I20260928 22:49:15.407128 21444 tablet_bootstrap.cc:492] T b239182a448f42e29bee1e97147c06a1 P 0ddf361e77374536897979c39e10bcb2: No bootstrap required, opened a new log
I20260928 22:49:15.407195 21444 ts_tablet_manager.cc:1397] T b239182a448f42e29bee1e97147c06a1 P 0ddf361e77374536897979c39e10bcb2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260928 22:49:15.408921 21444 raft_consensus.cc:348] T b239182a448f42e29bee1e97147c06a1 P 0ddf361e77374536897979c39e10bcb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564a049f0dfc4231b11f38c6d9d1453a" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 45529 } } peers { permanent_uuid: "94db7207ea574eb8a2796882d8fcfe29" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 35417 } } peers { permanent_uuid: "0ddf361e77374536897979c39e10bcb2" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 44087 } }
I20260928 22:49:15.409094 21444 raft_consensus.cc:374] T b239182a448f42e29bee1e97147c06a1 P 0ddf361e77374536897979c39e10bcb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:15.409119 21444 raft_consensus.cc:729] T b239182a448f42e29bee1e97147c06a1 P 0ddf361e77374536897979c39e10bcb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddf361e77374536897979c39e10bcb2, State: Initialized, Role: FOLLOWER
I20260928 22:49:15.409278 21444 consensus_queue.cc:260] T b239182a448f42e29bee1e97147c06a1 P 0ddf361e77374536897979c39e10bcb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564a049f0dfc4231b11f38c6d9d1453a" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 45529 } } peers { permanent_uuid: "94db7207ea574eb8a2796882d8fcfe29" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 35417 } } peers { permanent_uuid: "0ddf361e77374536897979c39e10bcb2" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 44087 } }
I20260928 22:49:15.409586 21444 ts_tablet_manager.cc:1428] T b239182a448f42e29bee1e97147c06a1 P 0ddf361e77374536897979c39e10bcb2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:49:15.409700 21293 heartbeater.cc:503] Master 127.16.121.190:40673 was elected leader, sending a full tablet report...
W20260928 22:49:15.432242 21294 tablet.cc:2369] T b239182a448f42e29bee1e97147c06a1 P 0ddf361e77374536897979c39e10bcb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:49:15.476786 21161 tablet.cc:2369] T b239182a448f42e29bee1e97147c06a1 P 94db7207ea574eb8a2796882d8fcfe29: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:49:15.568367 21451 raft_consensus.cc:482] T b239182a448f42e29bee1e97147c06a1 P 0ddf361e77374536897979c39e10bcb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:49:15.568513 21451 raft_consensus.cc:504] T b239182a448f42e29bee1e97147c06a1 P 0ddf361e77374536897979c39e10bcb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564a049f0dfc4231b11f38c6d9d1453a" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 45529 } } peers { permanent_uuid: "94db7207ea574eb8a2796882d8fcfe29" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 35417 } } peers { permanent_uuid: "0ddf361e77374536897979c39e10bcb2" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 44087 } }
I20260928 22:49:15.568889 21451 leader_election.cc:288] T b239182a448f42e29bee1e97147c06a1 P 0ddf361e77374536897979c39e10bcb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 564a049f0dfc4231b11f38c6d9d1453a (127.16.121.131:45529), 94db7207ea574eb8a2796882d8fcfe29 (127.16.121.129:35417)
I20260928 22:49:15.573065 21440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:15.569190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:55713 (local address 127.16.121.129:35417)
0928 22:49:15.569246 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:15.569248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:15.569289 (+ 41us) server_negotiation.cc:407] Connection header received
0928 22:49:15.569320 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:15.569321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:15.569339 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:15.569384 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:49:15.569608 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:15.570361 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:15.571984 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:15.572637 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:15.572704 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:15.572719 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:15.572941 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 22:49:15.572989 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 22:49:15.573119 21442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:15.569228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:44553 (local address 127.16.121.131:45529)
0928 22:49:15.569319 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:15.569321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:15.569437 (+ 116us) server_negotiation.cc:407] Connection header received
0928 22:49:15.569469 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:15.569470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:15.569479 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:15.569538 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:49:15.569692 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:15.570421 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:15.572019 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:15.572593 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:15.572688 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:15.572699 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:15.573003 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 22:49:15.573046 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 22:49:15.573098 21295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:15.569117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:35417 (local address 127.16.121.130:55713)
0928 22:49:15.569177 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:15.569190 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:15.569240 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:15.569412 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:15.569413 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:15.569416 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:49:15.569495 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:15.569496 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:15.570482 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:15.570485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:15.571869 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:15.571873 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:15.572740 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:15.572741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:15.572845 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:15.572847 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 22:49:15.572926 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 22:49:15.573002 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":84}
I20260928 22:49:15.573107 21452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:15.569131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.131:45529 (local address 127.16.121.130:44553)
0928 22:49:15.569335 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:15.569344 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:15.569419 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:15.569557 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:15.569558 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:15.569560 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:49:15.569619 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:15.569621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:15.570503 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:15.570503 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:15.571869 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:15.571873 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:15.572740 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:15.572741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:15.572840 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:15.572845 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:49:15.572986 (+ 141us) client_negotiation.cc:240] Negotiation successful
0928 22:49:15.573020 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":45,"thread_start_us":129,"threads_started":1}
I20260928 22:49:15.573587 21380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b239182a448f42e29bee1e97147c06a1" candidate_uuid: "0ddf361e77374536897979c39e10bcb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "564a049f0dfc4231b11f38c6d9d1453a" is_pre_election: true
I20260928 22:49:15.573792 21380 raft_consensus.cc:2454] T b239182a448f42e29bee1e97147c06a1 P 564a049f0dfc4231b11f38c6d9d1453a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ddf361e77374536897979c39e10bcb2 in term 0.
I20260928 22:49:15.573745 21114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b239182a448f42e29bee1e97147c06a1" candidate_uuid: "0ddf361e77374536897979c39e10bcb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94db7207ea574eb8a2796882d8fcfe29" is_pre_election: true
I20260928 22:49:15.573948 21114 raft_consensus.cc:2454] T b239182a448f42e29bee1e97147c06a1 P 94db7207ea574eb8a2796882d8fcfe29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ddf361e77374536897979c39e10bcb2 in term 0.
I20260928 22:49:15.574070 21180 leader_election.cc:302] T b239182a448f42e29bee1e97147c06a1 P 0ddf361e77374536897979c39e10bcb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddf361e77374536897979c39e10bcb2, 564a049f0dfc4231b11f38c6d9d1453a; no voters:
I20260928 22:49:15.574240 21451 raft_consensus.cc:2781] T b239182a448f42e29bee1e97147c06a1 P 0ddf361e77374536897979c39e10bcb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:49:15.574308 21451 raft_consensus.cc:482] T b239182a448f42e29bee1e97147c06a1 P 0ddf361e77374536897979c39e10bcb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:49:15.574326 21451 raft_consensus.cc:3037] T b239182a448f42e29bee1e97147c06a1 P 0ddf361e77374536897979c39e10bcb2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:15.575234 21451 raft_consensus.cc:504] T b239182a448f42e29bee1e97147c06a1 P 0ddf361e77374536897979c39e10bcb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564a049f0dfc4231b11f38c6d9d1453a" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 45529 } } peers { permanent_uuid: "94db7207ea574eb8a2796882d8fcfe29" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 35417 } } peers { permanent_uuid: "0ddf361e77374536897979c39e10bcb2" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 44087 } }
I20260928 22:49:15.575407 21451 leader_election.cc:288] T b239182a448f42e29bee1e97147c06a1 P 0ddf361e77374536897979c39e10bcb2 [CANDIDATE]: Term 1 election: Requested vote from peers 564a049f0dfc4231b11f38c6d9d1453a (127.16.121.131:45529), 94db7207ea574eb8a2796882d8fcfe29 (127.16.121.129:35417)
I20260928 22:49:15.575598 21114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b239182a448f42e29bee1e97147c06a1" candidate_uuid: "0ddf361e77374536897979c39e10bcb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94db7207ea574eb8a2796882d8fcfe29"
I20260928 22:49:15.575697 21114 raft_consensus.cc:3037] T b239182a448f42e29bee1e97147c06a1 P 94db7207ea574eb8a2796882d8fcfe29 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:15.575717 21380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b239182a448f42e29bee1e97147c06a1" candidate_uuid: "0ddf361e77374536897979c39e10bcb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "564a049f0dfc4231b11f38c6d9d1453a"
I20260928 22:49:15.575819 21380 raft_consensus.cc:3037] T b239182a448f42e29bee1e97147c06a1 P 564a049f0dfc4231b11f38c6d9d1453a [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:15.576574 21114 raft_consensus.cc:2454] T b239182a448f42e29bee1e97147c06a1 P 94db7207ea574eb8a2796882d8fcfe29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ddf361e77374536897979c39e10bcb2 in term 1.
I20260928 22:49:15.576640 21380 raft_consensus.cc:2454] T b239182a448f42e29bee1e97147c06a1 P 564a049f0dfc4231b11f38c6d9d1453a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ddf361e77374536897979c39e10bcb2 in term 1.
I20260928 22:49:15.576776 21181 leader_election.cc:302] T b239182a448f42e29bee1e97147c06a1 P 0ddf361e77374536897979c39e10bcb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ddf361e77374536897979c39e10bcb2, 94db7207ea574eb8a2796882d8fcfe29; no voters:
I20260928 22:49:15.576996 21451 raft_consensus.cc:2781] T b239182a448f42e29bee1e97147c06a1 P 0ddf361e77374536897979c39e10bcb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:49:15.577189 21451 raft_consensus.cc:686] T b239182a448f42e29bee1e97147c06a1 P 0ddf361e77374536897979c39e10bcb2 [term 1 LEADER]: Becoming Leader. State: Replica: 0ddf361e77374536897979c39e10bcb2, State: Running, Role: LEADER
I20260928 22:49:15.577456 21451 consensus_queue.cc:237] T b239182a448f42e29bee1e97147c06a1 P 0ddf361e77374536897979c39e10bcb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564a049f0dfc4231b11f38c6d9d1453a" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 45529 } } peers { permanent_uuid: "94db7207ea574eb8a2796882d8fcfe29" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 35417 } } peers { permanent_uuid: "0ddf361e77374536897979c39e10bcb2" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 44087 } }
I20260928 22:49:15.578588 20970 catalog_manager.cc:5495] T b239182a448f42e29bee1e97147c06a1 P 0ddf361e77374536897979c39e10bcb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ddf361e77374536897979c39e10bcb2 (127.16.121.130). New cstate: current_term: 1 leader_uuid: "0ddf361e77374536897979c39e10bcb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564a049f0dfc4231b11f38c6d9d1453a" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 45529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94db7207ea574eb8a2796882d8fcfe29" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 35417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ddf361e77374536897979c39e10bcb2" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 44087 } health_report { overall_health: HEALTHY } } }
W20260928 22:49:15.600769 21427 tablet.cc:2369] T b239182a448f42e29bee1e97147c06a1 P 564a049f0dfc4231b11f38c6d9d1453a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:49:15.625725 21441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:15.622548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44876 (local address 127.16.121.130:44087)
0928 22:49:15.622627 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:15.622630 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:15.622639 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:49:15.622687 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:15.622688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:15.622710 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:15.622870 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:49:15.623099 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:15.623715 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:15.624637 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:15.624859 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:15.624951 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:15.625178 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:49:15.625408 (+ 230us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:49:15.625557 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:15.625578 (+ 21us) server_negotiation.cc:299] Negotiation successful
0928 22:49:15.625602 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 22:49:15.628249 21114 raft_consensus.cc:1261] T b239182a448f42e29bee1e97147c06a1 P 94db7207ea574eb8a2796882d8fcfe29 [term 1 FOLLOWER]: Refusing update from remote peer 0ddf361e77374536897979c39e10bcb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:49:15.628252 21380 raft_consensus.cc:1261] T b239182a448f42e29bee1e97147c06a1 P 564a049f0dfc4231b11f38c6d9d1453a [term 1 FOLLOWER]: Refusing update from remote peer 0ddf361e77374536897979c39e10bcb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:49:15.628710 21453 consensus_queue.cc:1035] T b239182a448f42e29bee1e97147c06a1 P 0ddf361e77374536897979c39e10bcb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94db7207ea574eb8a2796882d8fcfe29" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 35417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:49:15.628932 21451 consensus_queue.cc:1035] T b239182a448f42e29bee1e97147c06a1 P 0ddf361e77374536897979c39e10bcb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "564a049f0dfc4231b11f38c6d9d1453a" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 45529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:49:15.631742 21457 mvcc.cc:205] Tried to move back new op lower bound from 7334444055049367552 to 7334444054846099456. Current Snapshot: MvccSnapshot[applied={T|T < 7334444055049367552}]
I20260928 22:49:15.632073 21458 mvcc.cc:205] Tried to move back new op lower bound from 7334444055049367552 to 7334444054846099456. Current Snapshot: MvccSnapshot[applied={T|T < 7334444055049367552}]
I20260928 22:49:15.632638 21456 mvcc.cc:205] Tried to move back new op lower bound from 7334444055049367552 to 7334444054846099456. Current Snapshot: MvccSnapshot[applied={T|T < 7334444055049367552 or (T in {7334444055049367552})}]
I20260928 22:49:15.634186 20969 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:49:15.634322 20969 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:49:15.635169 20969 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5ffa3da6cf9d48b3bc503fa883142b0b: Sending DeleteTablet for 3 replicas of tablet b239182a448f42e29bee1e97147c06a1
I20260928 22:49:15.635551 21227 tablet_service.cc:1514] Processing DeleteTablet for tablet b239182a448f42e29bee1e97147c06a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:49:15 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:44862
I20260928 22:49:15.635548 21094 tablet_service.cc:1514] Processing DeleteTablet for tablet b239182a448f42e29bee1e97147c06a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:49:15 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:37880
I20260928 22:49:15.635810 21360 tablet_service.cc:1514] Processing DeleteTablet for tablet b239182a448f42e29bee1e97147c06a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:49:15 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:44864
I20260928 22:49:15.636025 21468 tablet_replica.cc:330] T b239182a448f42e29bee1e97147c06a1 P 564a049f0dfc4231b11f38c6d9d1453a: stopping tablet replica
I20260928 22:49:15.636139 21468 raft_consensus.cc:2229] T b239182a448f42e29bee1e97147c06a1 P 564a049f0dfc4231b11f38c6d9d1453a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:49:15.636257 21468 raft_consensus.cc:2258] T b239182a448f42e29bee1e97147c06a1 P 564a049f0dfc4231b11f38c6d9d1453a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:49:15.636404 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 21032
I20260928 22:49:15.638476 21467 tablet_replica.cc:330] T b239182a448f42e29bee1e97147c06a1 P 0ddf361e77374536897979c39e10bcb2: stopping tablet replica
I20260928 22:49:15.638586 21467 raft_consensus.cc:2229] T b239182a448f42e29bee1e97147c06a1 P 0ddf361e77374536897979c39e10bcb2 [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:49:15.638598 21468 ts_tablet_manager.cc:1905] T b239182a448f42e29bee1e97147c06a1 P 564a049f0dfc4231b11f38c6d9d1453a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:49:15.638695 21467 raft_consensus.cc:2258] T b239182a448f42e29bee1e97147c06a1 P 0ddf361e77374536897979c39e10bcb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:49:15.639199 21467 ts_tablet_manager.cc:1905] T b239182a448f42e29bee1e97147c06a1 P 0ddf361e77374536897979c39e10bcb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:49:15.640426 21468 ts_tablet_manager.cc:1918] T b239182a448f42e29bee1e97147c06a1 P 564a049f0dfc4231b11f38c6d9d1453a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:49:15.640511 21468 log.cc:1196] T b239182a448f42e29bee1e97147c06a1 P 564a049f0dfc4231b11f38c6d9d1453a: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-2/wal/wals/b239182a448f42e29bee1e97147c06a1
I20260928 22:49:15.640827 21468 ts_tablet_manager.cc:1939] T b239182a448f42e29bee1e97147c06a1 P 564a049f0dfc4231b11f38c6d9d1453a: Deleting consensus metadata
I20260928 22:49:15.640925 21467 ts_tablet_manager.cc:1918] T b239182a448f42e29bee1e97147c06a1 P 0ddf361e77374536897979c39e10bcb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:49:15.641000 21467 log.cc:1196] T b239182a448f42e29bee1e97147c06a1 P 0ddf361e77374536897979c39e10bcb2: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790635733329745-16870-0/minicluster-data/ts-1/wal/wals/b239182a448f42e29bee1e97147c06a1
I20260928 22:49:15.641268 21467 ts_tablet_manager.cc:1939] T b239182a448f42e29bee1e97147c06a1 P 0ddf361e77374536897979c39e10bcb2: Deleting consensus metadata
I20260928 22:49:15.641352 20954 catalog_manager.cc:4841] TS 564a049f0dfc4231b11f38c6d9d1453a (127.16.121.131:45529): tablet b239182a448f42e29bee1e97147c06a1 (table test-table [id=6d1fe742d6f24e1197378a6c2b8207cd]) successfully deleted
I20260928 22:49:15.641880 20956 catalog_manager.cc:4841] TS 0ddf361e77374536897979c39e10bcb2 (127.16.121.130:44087): tablet b239182a448f42e29bee1e97147c06a1 (table test-table [id=6d1fe742d6f24e1197378a6c2b8207cd]) successfully deleted
W20260928 22:49:15.644425 20955 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.121.129:35417 (error 108)
I20260928 22:49:15.645011 21437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:15.644861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:35417 (local address 127.0.0.1:37896)
0928 22:49:15.644930 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:15.644973 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.129:35417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 22:49:15.645033 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 21165
W20260928 22:49:15.645148 20955 catalog_manager.cc:4584] TS 94db7207ea574eb8a2796882d8fcfe29 (127.16.121.129:35417): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b239182a448f42e29bee1e97147c06a1: Network error: Client connection negotiation failed: client connection to 127.16.121.129:35417: connect: Connection refused (error 111)
I20260928 22:49:15.650933 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 21298
I20260928 22:49:15.656020 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 20939
2026-09-28T22:49:15Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2245 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T22:49:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:49:15Z Disabled control of system clock
I20260928 22:49:15.685494 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:36051
--webserver_interface=127.16.121.190
--webserver_port=0
--builtin_ntp_servers=127.16.121.148:46163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:36051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 22:49:15.778730 21473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:15.779101 21473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:15.779137 21473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:15.781117 21473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:49:15.781188 21473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:15.781203 21473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:15.781215 21473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:49:15.781229 21473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:49:15.783285 21473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:46163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:36051
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:36051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.121.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:15.783579 21473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:15.783849 21473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:15.791230 21481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:15.791234 21479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:15.791234 21478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:15.791424 21473 server_base.cc:1034] running on GCE node
I20260928 22:49:15.791826 21473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:15.792181 21473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:15.793408 21473 hybrid_clock.cc:648] HybridClock initialized: now 1790635755793334 us; error 52 us; skew 500 ppm
I20260928 22:49:15.794957 21473 webserver.cc:460] Webserver started at http://127.16.121.190:38319/ using document root <none> and password file <none>
I20260928 22:49:15.795248 21473 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:15.795301 21473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:15.795439 21473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:15.796641 21473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "7ce941a5463f472eaca59401f0f93e97"
format_stamp: "Formatted at 2026-09-28 22:49:15 on dist-test-slave-56vw"
server_key: "d91ad990c039c36adf5fbbad6d644ec5"
server_key_iv: "0771a87f21a1f3591ec86cee15e909a9"
server_key_version: "encryptionkey@0"
I20260928 22:49:15.797118 21473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "7ce941a5463f472eaca59401f0f93e97"
format_stamp: "Formatted at 2026-09-28 22:49:15 on dist-test-slave-56vw"
server_key: "d91ad990c039c36adf5fbbad6d644ec5"
server_key_iv: "0771a87f21a1f3591ec86cee15e909a9"
server_key_version: "encryptionkey@0"
I20260928 22:49:15.798712 21473 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:49:15.799788 21487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:15.800050 21473 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:49:15.800156 21473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "7ce941a5463f472eaca59401f0f93e97"
format_stamp: "Formatted at 2026-09-28 22:49:15 on dist-test-slave-56vw"
server_key: "d91ad990c039c36adf5fbbad6d644ec5"
server_key_iv: "0771a87f21a1f3591ec86cee15e909a9"
server_key_version: "encryptionkey@0"
I20260928 22:49:15.800248 21473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:15.804498 21473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:15.804826 21473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:15.805047 21473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:15.809774 21473 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.190:36051
I20260928 22:49:15.809821 21539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.190:36051 every 8 connection(s)
I20260928 22:49:15.810340 21473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:49:15.810796 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 21473
I20260928 22:49:15.810889 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:49:15.811164 16870 external_mini_cluster.cc:1428] Setting key f330f3baea13e940f5759187474e64ef
I20260928 22:49:15.811317 21540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:15.814760 21543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:15.812065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34718 (local address 127.16.121.190:36051)
0928 22:49:15.812315 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:15.812325 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:15.812711 (+ 386us) server_negotiation.cc:407] Connection header received
0928 22:49:15.813282 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:15.813290 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:15.813339 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:15.813514 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:15.814267 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:15.814271 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:15.814272 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:49:15.814277 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:15.814330 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:49:15.814341 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:15.814421 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:15.814488 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 22:49:15.814554 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":89,"threads_started":1}
I20260928 22:49:15.815341 21540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ce941a5463f472eaca59401f0f93e97: Bootstrap starting.
I20260928 22:49:15.816036 21540 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ce941a5463f472eaca59401f0f93e97: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:15.816277 21540 log.cc:824] T 00000000000000000000000000000000 P 7ce941a5463f472eaca59401f0f93e97: Log is configured to *not* fsync() on all Append() calls
I20260928 22:49:15.817130 21540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ce941a5463f472eaca59401f0f93e97: No bootstrap required, opened a new log
I20260928 22:49:15.818763 21540 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ce941a5463f472eaca59401f0f93e97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ce941a5463f472eaca59401f0f93e97" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 36051 } }
I20260928 22:49:15.818921 21540 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ce941a5463f472eaca59401f0f93e97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:15.818953 21540 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ce941a5463f472eaca59401f0f93e97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ce941a5463f472eaca59401f0f93e97, State: Initialized, Role: FOLLOWER
I20260928 22:49:15.819108 21540 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ce941a5463f472eaca59401f0f93e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ce941a5463f472eaca59401f0f93e97" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 36051 } }
I20260928 22:49:15.819196 21540 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ce941a5463f472eaca59401f0f93e97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:49:15.819219 21540 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ce941a5463f472eaca59401f0f93e97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:49:15.819250 21540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ce941a5463f472eaca59401f0f93e97 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:15.819823 21540 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ce941a5463f472eaca59401f0f93e97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ce941a5463f472eaca59401f0f93e97" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 36051 } }
I20260928 22:49:15.819919 21540 leader_election.cc:302] T 00000000000000000000000000000000 P 7ce941a5463f472eaca59401f0f93e97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ce941a5463f472eaca59401f0f93e97; no voters:
I20260928 22:49:15.820190 21540 leader_election.cc:288] T 00000000000000000000000000000000 P 7ce941a5463f472eaca59401f0f93e97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:49:15.820636 21545 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ce941a5463f472eaca59401f0f93e97 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:49:15.820786 21540 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ce941a5463f472eaca59401f0f93e97 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:49:15.820827 21545 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ce941a5463f472eaca59401f0f93e97 [term 1 LEADER]: Becoming Leader. State: Replica: 7ce941a5463f472eaca59401f0f93e97, State: Running, Role: LEADER
I20260928 22:49:15.820930 21545 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ce941a5463f472eaca59401f0f93e97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ce941a5463f472eaca59401f0f93e97" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 36051 } }
I20260928 22:49:15.821313 21547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ce941a5463f472eaca59401f0f93e97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ce941a5463f472eaca59401f0f93e97. Latest consensus state: current_term: 1 leader_uuid: "7ce941a5463f472eaca59401f0f93e97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ce941a5463f472eaca59401f0f93e97" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 36051 } } }
I20260928 22:49:15.821430 21547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ce941a5463f472eaca59401f0f93e97 [sys.catalog]: This master's current role is: LEADER
I20260928 22:49:15.821305 21546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ce941a5463f472eaca59401f0f93e97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ce941a5463f472eaca59401f0f93e97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ce941a5463f472eaca59401f0f93e97" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 36051 } } }
I20260928 22:49:15.821605 21546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ce941a5463f472eaca59401f0f93e97 [sys.catalog]: This master's current role is: LEADER
I20260928 22:49:15.821722 21554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:49:15.822261 21554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:49:15.823896 21554 catalog_manager.cc:1347] Generated new cluster ID: b3abc0d44b3a46299e1a0b3c81e6d7d7
I20260928 22:49:15.823961 21554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:49:15.830052 21554 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:49:15.830286 21554 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:49:15.840497 21554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ce941a5463f472eaca59401f0f93e97: Generated new TSK 0
I20260928 22:49:15.840754 21554 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:49:15.846105 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.129:0
--local_ip_for_outbound_sockets=127.16.121.129
--webserver_interface=127.16.121.129
--webserver_port=0
--tserver_master_addrs=127.16.121.190:36051
--builtin_ntp_servers=127.16.121.148:46163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 22:49:15.935951 21564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:15.936195 21564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:15.936241 21564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:15.937489 21564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:15.937552 21564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:15.937597 21564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.129
I20260928 22:49:15.939062 21564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:46163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.121.129
--webserver_port=0
--tserver_master_addrs=127.16.121.190:36051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.129
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:15.939368 21564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:15.939684 21564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:15.945725 21572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:15.945725 21570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:15.945749 21569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:15.946175 21564 server_base.cc:1034] running on GCE node
I20260928 22:49:15.946393 21564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:15.946656 21564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:15.947861 21564 hybrid_clock.cc:648] HybridClock initialized: now 1790635755947832 us; error 45 us; skew 500 ppm
I20260928 22:49:15.949460 21564 webserver.cc:460] Webserver started at http://127.16.121.129:40995/ using document root <none> and password file <none>
I20260928 22:49:15.949746 21564 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:15.949800 21564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:15.949985 21564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:15.951350 21564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-0/data/instance:
uuid: "a995b5225ea94d9ca0e29f51d924b8e4"
format_stamp: "Formatted at 2026-09-28 22:49:15 on dist-test-slave-56vw"
server_key: "080884c22cbbc47e32aaaac2dee68688"
server_key_iv: "913cb4eee71f03384fd2eede44db7f5d"
server_key_version: "encryptionkey@0"
I20260928 22:49:15.951826 21564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance:
uuid: "a995b5225ea94d9ca0e29f51d924b8e4"
format_stamp: "Formatted at 2026-09-28 22:49:15 on dist-test-slave-56vw"
server_key: "080884c22cbbc47e32aaaac2dee68688"
server_key_iv: "913cb4eee71f03384fd2eede44db7f5d"
server_key_version: "encryptionkey@0"
I20260928 22:49:15.953429 21564 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:49:15.954314 21578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:15.954478 21564 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:15.954599 21564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-0/wal
uuid: "a995b5225ea94d9ca0e29f51d924b8e4"
format_stamp: "Formatted at 2026-09-28 22:49:15 on dist-test-slave-56vw"
server_key: "080884c22cbbc47e32aaaac2dee68688"
server_key_iv: "913cb4eee71f03384fd2eede44db7f5d"
server_key_version: "encryptionkey@0"
I20260928 22:49:15.954684 21564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:15.959023 21564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:15.959410 21564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:15.959573 21564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:15.959875 21564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:49:15.960278 21564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:49:15.960316 21564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:15.960341 21564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:49:15.960389 21564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:15.966825 21564 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.129:33305
I20260928 22:49:15.966950 21691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.129:33305 every 8 connection(s)
I20260928 22:49:15.967264 21564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
I20260928 22:49:15.970475 21543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:15.968042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:50157 (local address 127.16.121.190:36051)
0928 22:49:15.968103 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:15.968105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:15.968313 (+ 208us) server_negotiation.cc:407] Connection header received
0928 22:49:15.968900 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:15.968902 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:15.968907 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:15.968963 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:15.970158 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:15.970159 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:15.970159 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:49:15.970163 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:15.970178 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:49:15.970182 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:15.970246 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:15.970407 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 22:49:15.970422 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 22:49:15.970523 21694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:15.968018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:36051 (local address 127.16.121.129:50157)
0928 22:49:15.968206 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:15.968241 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:15.968796 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:15.969019 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:15.969022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:15.969030 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:15.969985 (+ 955us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:49:15.969986 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:15.970018 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:49:15.970047 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:15.970248 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:15.970251 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:15.970253 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 22:49:15.970307 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 22:49:15.970354 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":99,"threads_started":1}
I20260928 22:49:15.970988 21692 heartbeater.cc:347] Connected to a master server at 127.16.121.190:36051
I20260928 22:49:15.971123 21692 heartbeater.cc:464] Registering TS with master...
I20260928 22:49:15.971302 21692 heartbeater.cc:511] Master 127.16.121.190:36051 requested a full tablet report, sending...
I20260928 22:49:15.971859 21504 ts_manager.cc:194] Registered new tserver with Master: a995b5225ea94d9ca0e29f51d924b8e4 (127.16.121.129:33305)
I20260928 22:49:15.971869 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 21564
I20260928 22:49:15.971980 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance
I20260928 22:49:15.972210 16870 external_mini_cluster.cc:1428] Setting key 2222aee80691ee54188080e8f4ccaca2
I20260928 22:49:15.973222 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.130:0
--local_ip_for_outbound_sockets=127.16.121.130
--webserver_interface=127.16.121.130
--webserver_port=0
--tserver_master_addrs=127.16.121.190:36051
--builtin_ntp_servers=127.16.121.148:46163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 22:49:16.058251 21695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:16.058537 21695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:16.058578 21695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:16.059975 21695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:16.060021 21695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:16.060050 21695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.130
I20260928 22:49:16.061405 21695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:46163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.121.130
--webserver_port=0
--tserver_master_addrs=127.16.121.190:36051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.130
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:16.061636 21695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:16.061901 21695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:16.067269 21703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:16.067458 21695 server_base.cc:1034] running on GCE node
W20260928 22:49:16.067239 21700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:16.067241 21701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:16.067816 21695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:16.068086 21695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:16.069239 21695 hybrid_clock.cc:648] HybridClock initialized: now 1790635756069218 us; error 43 us; skew 500 ppm
I20260928 22:49:16.070708 21695 webserver.cc:460] Webserver started at http://127.16.121.130:43215/ using document root <none> and password file <none>
I20260928 22:49:16.070989 21695 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:16.071086 21695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:16.071245 21695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:16.072361 21695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-1/data/instance:
uuid: "7e3833a70e3441148078ed102fcf7931"
format_stamp: "Formatted at 2026-09-28 22:49:16 on dist-test-slave-56vw"
server_key: "3aff5e0b74d2184a94f11fdb122c34af"
server_key_iv: "43577c430cb2fc7246b852dfa798d93c"
server_key_version: "encryptionkey@0"
I20260928 22:49:16.072751 21695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance:
uuid: "7e3833a70e3441148078ed102fcf7931"
format_stamp: "Formatted at 2026-09-28 22:49:16 on dist-test-slave-56vw"
server_key: "3aff5e0b74d2184a94f11fdb122c34af"
server_key_iv: "43577c430cb2fc7246b852dfa798d93c"
server_key_version: "encryptionkey@0"
I20260928 22:49:16.074239 21695 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:49:16.075177 21709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:16.075392 21695 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 22:49:16.075510 21695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-1/wal
uuid: "7e3833a70e3441148078ed102fcf7931"
format_stamp: "Formatted at 2026-09-28 22:49:16 on dist-test-slave-56vw"
server_key: "3aff5e0b74d2184a94f11fdb122c34af"
server_key_iv: "43577c430cb2fc7246b852dfa798d93c"
server_key_version: "encryptionkey@0"
I20260928 22:49:16.075600 21695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:16.079113 21695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:16.079411 21695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:16.079560 21695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:16.079787 21695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:49:16.080124 21695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:49:16.080179 21695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:16.080219 21695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:49:16.080245 21695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:16.086755 21695 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.130:32785
I20260928 22:49:16.086799 21822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.130:32785 every 8 connection(s)
I20260928 22:49:16.087275 21695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
I20260928 22:49:16.088742 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 21695
I20260928 22:49:16.088832 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance
I20260928 22:49:16.089064 16870 external_mini_cluster.cc:1428] Setting key 10d574215ef83260bedb35f138061e85
I20260928 22:49:16.090646 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.131:0
--local_ip_for_outbound_sockets=127.16.121.131
--webserver_interface=127.16.121.131
--webserver_port=0
--tserver_master_addrs=127.16.121.190:36051
--builtin_ntp_servers=127.16.121.148:46163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 22:49:16.091019 21543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:16.088260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:42953 (local address 127.16.121.190:36051)
0928 22:49:16.088305 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:16.088307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:16.088619 (+ 312us) server_negotiation.cc:407] Connection header received
0928 22:49:16.089346 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:16.089348 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:16.089355 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:16.089406 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:16.090679 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:16.090680 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:16.090681 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:49:16.090686 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:16.090703 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:49:16.090707 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:16.090787 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:16.090931 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 22:49:16.090947 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:49:16.091286 21825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:16.088273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:36051 (local address 127.16.121.130:42953)
0928 22:49:16.088486 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:16.088524 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:16.089198 (+ 674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:16.089487 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:16.089492 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:16.089503 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:16.090503 (+ 1000us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:49:16.090503 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:16.090534 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:49:16.090575 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:16.090850 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:16.090857 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:16.090860 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:49:16.090943 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 22:49:16.091012 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":86,"threads_started":1}
I20260928 22:49:16.091813 21823 heartbeater.cc:347] Connected to a master server at 127.16.121.190:36051
I20260928 22:49:16.091928 21823 heartbeater.cc:464] Registering TS with master...
I20260928 22:49:16.092111 21823 heartbeater.cc:511] Master 127.16.121.190:36051 requested a full tablet report, sending...
I20260928 22:49:16.092707 21504 ts_manager.cc:194] Registered new tserver with Master: 7e3833a70e3441148078ed102fcf7931 (127.16.121.130:32785)
W20260928 22:49:16.177711 21826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:16.177979 21826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:16.178021 21826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:16.179518 21826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:16.179584 21826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:16.179631 21826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.131
I20260928 22:49:16.181097 21826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:46163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.121.131
--webserver_port=0
--tserver_master_addrs=127.16.121.190:36051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.131
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:16.181384 21826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:16.181699 21826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:16.187243 21834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:16.187405 21826 server_base.cc:1034] running on GCE node
W20260928 22:49:16.187261 21831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:16.187243 21832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:16.187696 21826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:16.188017 21826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:16.189190 21826 hybrid_clock.cc:648] HybridClock initialized: now 1790635756189175 us; error 45 us; skew 500 ppm
I20260928 22:49:16.190608 21826 webserver.cc:460] Webserver started at http://127.16.121.131:33109/ using document root <none> and password file <none>
I20260928 22:49:16.190898 21826 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:16.190973 21826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:16.191151 21826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:16.192351 21826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-2/data/instance:
uuid: "83ce3c4dcf584675a8ee8c50c417e5fa"
format_stamp: "Formatted at 2026-09-28 22:49:16 on dist-test-slave-56vw"
server_key: "a3db73e6194684665234dc6d763bcdf4"
server_key_iv: "7fe41a96fcd59e97386b060662b2ffe4"
server_key_version: "encryptionkey@0"
I20260928 22:49:16.192737 21826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance:
uuid: "83ce3c4dcf584675a8ee8c50c417e5fa"
format_stamp: "Formatted at 2026-09-28 22:49:16 on dist-test-slave-56vw"
server_key: "a3db73e6194684665234dc6d763bcdf4"
server_key_iv: "7fe41a96fcd59e97386b060662b2ffe4"
server_key_version: "encryptionkey@0"
I20260928 22:49:16.194131 21826 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 22:49:16.195264 21840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:16.195566 21826 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:16.195678 21826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-2/wal
uuid: "83ce3c4dcf584675a8ee8c50c417e5fa"
format_stamp: "Formatted at 2026-09-28 22:49:16 on dist-test-slave-56vw"
server_key: "a3db73e6194684665234dc6d763bcdf4"
server_key_iv: "7fe41a96fcd59e97386b060662b2ffe4"
server_key_version: "encryptionkey@0"
I20260928 22:49:16.195771 21826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:16.199699 21826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:16.200012 21826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:16.200160 21826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:16.200387 21826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:49:16.200726 21826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:49:16.200779 21826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:16.200821 21826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:49:16.200848 21826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:16.206939 21826 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.131:38749
I20260928 22:49:16.206996 21953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.131:38749 every 8 connection(s)
I20260928 22:49:16.207356 21826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
I20260928 22:49:16.210668 21543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:16.208120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:41879 (local address 127.16.121.190:36051)
0928 22:49:16.208165 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:16.208169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:16.208415 (+ 246us) server_negotiation.cc:407] Connection header received
0928 22:49:16.208961 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:16.208962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:16.208968 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:16.209039 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:16.210377 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:16.210379 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:16.210379 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:49:16.210384 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:16.210400 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:49:16.210404 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:16.210464 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:16.210597 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 22:49:16.210612 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 22:49:16.210904 21956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:16.208076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:36051 (local address 127.16.121.131:41879)
0928 22:49:16.208259 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:16.208302 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:16.208834 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:16.209160 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:16.209164 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:16.209181 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:16.210204 (+ 1023us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:49:16.210205 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:16.210236 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:49:16.210269 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:16.210491 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:16.210504 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:16.210507 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:49:16.210587 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 22:49:16.210644 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
I20260928 22:49:16.211481 21954 heartbeater.cc:347] Connected to a master server at 127.16.121.190:36051
I20260928 22:49:16.211580 21954 heartbeater.cc:464] Registering TS with master...
I20260928 22:49:16.211700 21954 heartbeater.cc:511] Master 127.16.121.190:36051 requested a full tablet report, sending...
I20260928 22:49:16.212052 21504 ts_manager.cc:194] Registered new tserver with Master: 83ce3c4dcf584675a8ee8c50c417e5fa (127.16.121.131:38749)
I20260928 22:49:16.216327 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 21826
I20260928 22:49:16.216466 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance
I20260928 22:49:16.216653 16870 external_mini_cluster.cc:1428] Setting key 89f159cc336cae4c781ef6475c11e7de
I20260928 22:49:16.217689 16870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:49:16.220561 21543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:16.219208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34726 (local address 127.16.121.190:36051)
0928 22:49:16.219250 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:16.219252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:16.219412 (+ 160us) server_negotiation.cc:407] Connection header received
0928 22:49:16.219448 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:16.219448 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:16.219454 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:16.219576 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:16.220259 (+ 683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:16.220260 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:16.220260 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:49:16.220264 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:16.220277 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:49:16.220280 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:16.220326 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:16.220360 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 22:49:16.220372 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 22:49:16.223358 21504 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:49:16.225067 21504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:49:16.229928 21966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:16.228104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41122 (local address 127.16.121.130:32785)
0928 22:49:16.228258 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:16.228264 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:16.228275 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:49:16.228305 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:16.228308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:16.228320 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:16.228939 (+ 619us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:16.229588 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:16.229595 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:16.229595 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:49:16.229599 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:16.229630 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:49:16.229663 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:16.229746 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:16.229822 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 22:49:16.229836 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260928 22:49:16.229928 21967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:16.228311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43726 (local address 127.16.121.129:33305)
0928 22:49:16.228488 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:16.228493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:16.228503 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:49:16.228529 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:16.228531 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:16.228541 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:16.228879 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:16.229527 (+ 648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:16.229536 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:16.229537 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:49:16.229542 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:16.229582 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:49:16.229614 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:16.229683 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:16.229817 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 22:49:16.229834 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":106,"threads_started":1}
I20260928 22:49:16.229914 21965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:16.228080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:33305 (local address 127.0.0.1:43726)
0928 22:49:16.228331 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:16.228341 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:16.228386 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:16.228633 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:16.228634 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:16.228638 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:16.229380 (+ 742us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:49:16.229381 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:16.229399 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:49:16.229434 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:16.229709 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:16.229716 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:16.229719 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:49:16.229795 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 22:49:16.229810 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":80,"threads_started":1}
I20260928 22:49:16.229919 21962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:16.227838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.130:32785 (local address 127.0.0.1:41122)
0928 22:49:16.228044 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:16.228056 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:16.228108 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:16.228613 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:16.228621 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:16.228628 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:16.229460 (+ 832us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:49:16.229461 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:16.229467 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:49:16.229500 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:16.229743 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:16.229743 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:16.229744 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 22:49:16.229808 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 22:49:16.229823 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":96,"threads_started":1}
I20260928 22:49:16.230664 21963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:16.227780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.131:38749 (local address 127.0.0.1:47538)
0928 22:49:16.227999 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:16.228043 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:16.228102 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:16.228744 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:16.228745 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:16.228747 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:16.229609 (+ 862us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:49:16.229609 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:16.229614 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:49:16.229622 (+ 8us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:16.230524 (+ 902us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:16.230525 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:16.230525 (+ 0us) client_negotiation.cc:769] Sending connection context
0928 22:49:16.230585 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 22:49:16.230599 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":98,"threads_started":1}
I20260928 22:49:16.230693 21964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:16.227926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47538 (local address 127.16.121.131:38749)
0928 22:49:16.228309 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:16.228313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:16.228322 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:49:16.228348 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:16.228350 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:16.228359 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:16.228430 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:16.230378 (+ 1948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:16.230386 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:16.230387 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:49:16.230391 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:16.230424 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:49:16.230443 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:16.230527 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:16.230596 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 22:49:16.230611 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":82,"threads_started":1}
I20260928 22:49:16.230726 21626 tablet_service.cc:1467] Processing CreateTablet for tablet c0b87dc108f4403f8aa5dee538564479 (DEFAULT_TABLE table=test-table [id=da8dc38055ee4fb7855136f663657bed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:49:16.230727 21757 tablet_service.cc:1467] Processing CreateTablet for tablet c0b87dc108f4403f8aa5dee538564479 (DEFAULT_TABLE table=test-table [id=da8dc38055ee4fb7855136f663657bed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:49:16.231135 21626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0b87dc108f4403f8aa5dee538564479. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:16.231135 21757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0b87dc108f4403f8aa5dee538564479. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:16.231556 21888 tablet_service.cc:1467] Processing CreateTablet for tablet c0b87dc108f4403f8aa5dee538564479 (DEFAULT_TABLE table=test-table [id=da8dc38055ee4fb7855136f663657bed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:49:16.231894 21888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0b87dc108f4403f8aa5dee538564479. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:16.234495 21968 tablet_bootstrap.cc:492] T c0b87dc108f4403f8aa5dee538564479 P 83ce3c4dcf584675a8ee8c50c417e5fa: Bootstrap starting.
I20260928 22:49:16.234838 21969 tablet_bootstrap.cc:492] T c0b87dc108f4403f8aa5dee538564479 P 7e3833a70e3441148078ed102fcf7931: Bootstrap starting.
I20260928 22:49:16.235327 21968 tablet_bootstrap.cc:654] T c0b87dc108f4403f8aa5dee538564479 P 83ce3c4dcf584675a8ee8c50c417e5fa: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:16.235533 21969 tablet_bootstrap.cc:654] T c0b87dc108f4403f8aa5dee538564479 P 7e3833a70e3441148078ed102fcf7931: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:16.235683 21968 log.cc:824] T c0b87dc108f4403f8aa5dee538564479 P 83ce3c4dcf584675a8ee8c50c417e5fa: Log is configured to *not* fsync() on all Append() calls
I20260928 22:49:16.235867 21969 log.cc:824] T c0b87dc108f4403f8aa5dee538564479 P 7e3833a70e3441148078ed102fcf7931: Log is configured to *not* fsync() on all Append() calls
I20260928 22:49:16.236258 21970 tablet_bootstrap.cc:492] T c0b87dc108f4403f8aa5dee538564479 P a995b5225ea94d9ca0e29f51d924b8e4: Bootstrap starting.
I20260928 22:49:16.236521 21968 tablet_bootstrap.cc:492] T c0b87dc108f4403f8aa5dee538564479 P 83ce3c4dcf584675a8ee8c50c417e5fa: No bootstrap required, opened a new log
I20260928 22:49:16.236567 21968 ts_tablet_manager.cc:1397] T c0b87dc108f4403f8aa5dee538564479 P 83ce3c4dcf584675a8ee8c50c417e5fa: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:49:16.236727 21969 tablet_bootstrap.cc:492] T c0b87dc108f4403f8aa5dee538564479 P 7e3833a70e3441148078ed102fcf7931: No bootstrap required, opened a new log
I20260928 22:49:16.236785 21969 ts_tablet_manager.cc:1397] T c0b87dc108f4403f8aa5dee538564479 P 7e3833a70e3441148078ed102fcf7931: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:49:16.236946 21970 tablet_bootstrap.cc:654] T c0b87dc108f4403f8aa5dee538564479 P a995b5225ea94d9ca0e29f51d924b8e4: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:16.237267 21970 log.cc:824] T c0b87dc108f4403f8aa5dee538564479 P a995b5225ea94d9ca0e29f51d924b8e4: Log is configured to *not* fsync() on all Append() calls
I20260928 22:49:16.238080 21970 tablet_bootstrap.cc:492] T c0b87dc108f4403f8aa5dee538564479 P a995b5225ea94d9ca0e29f51d924b8e4: No bootstrap required, opened a new log
I20260928 22:49:16.238142 21970 ts_tablet_manager.cc:1397] T c0b87dc108f4403f8aa5dee538564479 P a995b5225ea94d9ca0e29f51d924b8e4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:49:16.239280 21969 raft_consensus.cc:348] T c0b87dc108f4403f8aa5dee538564479 P 7e3833a70e3441148078ed102fcf7931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ce3c4dcf584675a8ee8c50c417e5fa" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 38749 } } peers { permanent_uuid: "7e3833a70e3441148078ed102fcf7931" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32785 } } peers { permanent_uuid: "a995b5225ea94d9ca0e29f51d924b8e4" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33305 } }
I20260928 22:49:16.239205 21968 raft_consensus.cc:348] T c0b87dc108f4403f8aa5dee538564479 P 83ce3c4dcf584675a8ee8c50c417e5fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ce3c4dcf584675a8ee8c50c417e5fa" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 38749 } } peers { permanent_uuid: "7e3833a70e3441148078ed102fcf7931" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32785 } } peers { permanent_uuid: "a995b5225ea94d9ca0e29f51d924b8e4" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33305 } }
I20260928 22:49:16.239475 21969 raft_consensus.cc:374] T c0b87dc108f4403f8aa5dee538564479 P 7e3833a70e3441148078ed102fcf7931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:16.239496 21968 raft_consensus.cc:374] T c0b87dc108f4403f8aa5dee538564479 P 83ce3c4dcf584675a8ee8c50c417e5fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:16.239532 21969 raft_consensus.cc:729] T c0b87dc108f4403f8aa5dee538564479 P 7e3833a70e3441148078ed102fcf7931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e3833a70e3441148078ed102fcf7931, State: Initialized, Role: FOLLOWER
I20260928 22:49:16.239550 21968 raft_consensus.cc:729] T c0b87dc108f4403f8aa5dee538564479 P 83ce3c4dcf584675a8ee8c50c417e5fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83ce3c4dcf584675a8ee8c50c417e5fa, State: Initialized, Role: FOLLOWER
I20260928 22:49:16.239660 21969 consensus_queue.cc:260] T c0b87dc108f4403f8aa5dee538564479 P 7e3833a70e3441148078ed102fcf7931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ce3c4dcf584675a8ee8c50c417e5fa" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 38749 } } peers { permanent_uuid: "7e3833a70e3441148078ed102fcf7931" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32785 } } peers { permanent_uuid: "a995b5225ea94d9ca0e29f51d924b8e4" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33305 } }
I20260928 22:49:16.239693 21968 consensus_queue.cc:260] T c0b87dc108f4403f8aa5dee538564479 P 83ce3c4dcf584675a8ee8c50c417e5fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ce3c4dcf584675a8ee8c50c417e5fa" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 38749 } } peers { permanent_uuid: "7e3833a70e3441148078ed102fcf7931" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32785 } } peers { permanent_uuid: "a995b5225ea94d9ca0e29f51d924b8e4" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33305 } }
I20260928 22:49:16.239997 21968 ts_tablet_manager.cc:1428] T c0b87dc108f4403f8aa5dee538564479 P 83ce3c4dcf584675a8ee8c50c417e5fa: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 22:49:16.239998 21969 ts_tablet_manager.cc:1428] T c0b87dc108f4403f8aa5dee538564479 P 7e3833a70e3441148078ed102fcf7931: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 22:49:16.240144 21823 heartbeater.cc:503] Master 127.16.121.190:36051 was elected leader, sending a full tablet report...
I20260928 22:49:16.240180 21954 heartbeater.cc:503] Master 127.16.121.190:36051 was elected leader, sending a full tablet report...
I20260928 22:49:16.241379 21970 raft_consensus.cc:348] T c0b87dc108f4403f8aa5dee538564479 P a995b5225ea94d9ca0e29f51d924b8e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ce3c4dcf584675a8ee8c50c417e5fa" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 38749 } } peers { permanent_uuid: "7e3833a70e3441148078ed102fcf7931" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32785 } } peers { permanent_uuid: "a995b5225ea94d9ca0e29f51d924b8e4" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33305 } }
I20260928 22:49:16.241593 21970 raft_consensus.cc:374] T c0b87dc108f4403f8aa5dee538564479 P a995b5225ea94d9ca0e29f51d924b8e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:16.241641 21970 raft_consensus.cc:729] T c0b87dc108f4403f8aa5dee538564479 P a995b5225ea94d9ca0e29f51d924b8e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a995b5225ea94d9ca0e29f51d924b8e4, State: Initialized, Role: FOLLOWER
I20260928 22:49:16.241772 21970 consensus_queue.cc:260] T c0b87dc108f4403f8aa5dee538564479 P a995b5225ea94d9ca0e29f51d924b8e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ce3c4dcf584675a8ee8c50c417e5fa" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 38749 } } peers { permanent_uuid: "7e3833a70e3441148078ed102fcf7931" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32785 } } peers { permanent_uuid: "a995b5225ea94d9ca0e29f51d924b8e4" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33305 } }
I20260928 22:49:16.242080 21970 ts_tablet_manager.cc:1428] T c0b87dc108f4403f8aa5dee538564479 P a995b5225ea94d9ca0e29f51d924b8e4: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 22:49:16.242129 21692 heartbeater.cc:503] Master 127.16.121.190:36051 was elected leader, sending a full tablet report...
W20260928 22:49:16.338083 21824 tablet.cc:2369] T c0b87dc108f4403f8aa5dee538564479 P 7e3833a70e3441148078ed102fcf7931: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:49:16.458169 21955 tablet.cc:2369] T c0b87dc108f4403f8aa5dee538564479 P 83ce3c4dcf584675a8ee8c50c417e5fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:49:16.468102 21693 tablet.cc:2369] T c0b87dc108f4403f8aa5dee538564479 P a995b5225ea94d9ca0e29f51d924b8e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:49:16.468870 21976 raft_consensus.cc:482] T c0b87dc108f4403f8aa5dee538564479 P a995b5225ea94d9ca0e29f51d924b8e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:49:16.469012 21976 raft_consensus.cc:504] T c0b87dc108f4403f8aa5dee538564479 P a995b5225ea94d9ca0e29f51d924b8e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ce3c4dcf584675a8ee8c50c417e5fa" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 38749 } } peers { permanent_uuid: "7e3833a70e3441148078ed102fcf7931" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32785 } } peers { permanent_uuid: "a995b5225ea94d9ca0e29f51d924b8e4" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33305 } }
I20260928 22:49:16.469359 21976 leader_election.cc:288] T c0b87dc108f4403f8aa5dee538564479 P a995b5225ea94d9ca0e29f51d924b8e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83ce3c4dcf584675a8ee8c50c417e5fa (127.16.121.131:38749), 7e3833a70e3441148078ed102fcf7931 (127.16.121.130:32785)
I20260928 22:49:16.471078 21694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:16.469581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.131:38749 (local address 127.16.121.129:39545)
0928 22:49:16.469644 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:16.469657 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:16.469733 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:16.470011 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:16.470013 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:16.470015 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:16.470616 (+ 601us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:49:16.470617 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:16.470624 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:49:16.470637 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:16.470898 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:16.470899 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:16.470900 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 22:49:16.470957 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 22:49:16.470974 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 22:49:16.471078 21964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:16.469735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:39545 (local address 127.16.121.131:38749)
0928 22:49:16.469813 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:16.469863 (+ 50us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:16.469875 (+ 12us) server_negotiation.cc:407] Connection header received
0928 22:49:16.469911 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:16.469912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:16.469917 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:16.469990 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:16.470805 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:16.470806 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:16.470807 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:49:16.470811 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:16.470826 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:49:16.470830 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:16.470901 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:16.470960 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 22:49:16.470974 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 22:49:16.471105 21966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:16.469825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:47769 (local address 127.16.121.130:32785)
0928 22:49:16.469876 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:16.469878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:16.469970 (+ 92us) server_negotiation.cc:407] Connection header received
0928 22:49:16.470040 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:16.470041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:16.470047 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:16.470103 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:16.470811 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:16.470813 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:16.470813 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:49:16.470817 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:16.470833 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:49:16.470838 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:16.470906 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:16.471005 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 22:49:16.471018 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 22:49:16.471105 21977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:16.469749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.130:32785 (local address 127.16.121.129:47769)
0928 22:49:16.469891 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:16.469907 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:16.469977 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:16.470114 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:16.470115 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:16.470117 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:16.470665 (+ 548us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:49:16.470665 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:16.470673 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:49:16.470686 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:16.470924 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:16.470924 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:16.470925 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 22:49:16.470970 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 22:49:16.470984 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":77,"threads_started":1}
I20260928 22:49:16.471515 21908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0b87dc108f4403f8aa5dee538564479" candidate_uuid: "a995b5225ea94d9ca0e29f51d924b8e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83ce3c4dcf584675a8ee8c50c417e5fa" is_pre_election: true
I20260928 22:49:16.471689 21908 raft_consensus.cc:2454] T c0b87dc108f4403f8aa5dee538564479 P 83ce3c4dcf584675a8ee8c50c417e5fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a995b5225ea94d9ca0e29f51d924b8e4 in term 0.
I20260928 22:49:16.471728 21777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0b87dc108f4403f8aa5dee538564479" candidate_uuid: "a995b5225ea94d9ca0e29f51d924b8e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e3833a70e3441148078ed102fcf7931" is_pre_election: true
I20260928 22:49:16.471886 21777 raft_consensus.cc:2454] T c0b87dc108f4403f8aa5dee538564479 P 7e3833a70e3441148078ed102fcf7931 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a995b5225ea94d9ca0e29f51d924b8e4 in term 0.
I20260928 22:49:16.471948 21580 leader_election.cc:302] T c0b87dc108f4403f8aa5dee538564479 P a995b5225ea94d9ca0e29f51d924b8e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83ce3c4dcf584675a8ee8c50c417e5fa, a995b5225ea94d9ca0e29f51d924b8e4; no voters:
I20260928 22:49:16.472189 21976 raft_consensus.cc:2781] T c0b87dc108f4403f8aa5dee538564479 P a995b5225ea94d9ca0e29f51d924b8e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:49:16.472244 21976 raft_consensus.cc:482] T c0b87dc108f4403f8aa5dee538564479 P a995b5225ea94d9ca0e29f51d924b8e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:49:16.472332 21976 raft_consensus.cc:3037] T c0b87dc108f4403f8aa5dee538564479 P a995b5225ea94d9ca0e29f51d924b8e4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:16.473065 21976 raft_consensus.cc:504] T c0b87dc108f4403f8aa5dee538564479 P a995b5225ea94d9ca0e29f51d924b8e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ce3c4dcf584675a8ee8c50c417e5fa" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 38749 } } peers { permanent_uuid: "7e3833a70e3441148078ed102fcf7931" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32785 } } peers { permanent_uuid: "a995b5225ea94d9ca0e29f51d924b8e4" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33305 } }
I20260928 22:49:16.473217 21976 leader_election.cc:288] T c0b87dc108f4403f8aa5dee538564479 P a995b5225ea94d9ca0e29f51d924b8e4 [CANDIDATE]: Term 1 election: Requested vote from peers 83ce3c4dcf584675a8ee8c50c417e5fa (127.16.121.131:38749), 7e3833a70e3441148078ed102fcf7931 (127.16.121.130:32785)
I20260928 22:49:16.473392 21777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0b87dc108f4403f8aa5dee538564479" candidate_uuid: "a995b5225ea94d9ca0e29f51d924b8e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e3833a70e3441148078ed102fcf7931"
I20260928 22:49:16.473387 21908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0b87dc108f4403f8aa5dee538564479" candidate_uuid: "a995b5225ea94d9ca0e29f51d924b8e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83ce3c4dcf584675a8ee8c50c417e5fa"
I20260928 22:49:16.473471 21777 raft_consensus.cc:3037] T c0b87dc108f4403f8aa5dee538564479 P 7e3833a70e3441148078ed102fcf7931 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:16.473474 21908 raft_consensus.cc:3037] T c0b87dc108f4403f8aa5dee538564479 P 83ce3c4dcf584675a8ee8c50c417e5fa [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:16.474215 21777 raft_consensus.cc:2454] T c0b87dc108f4403f8aa5dee538564479 P 7e3833a70e3441148078ed102fcf7931 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a995b5225ea94d9ca0e29f51d924b8e4 in term 1.
I20260928 22:49:16.474220 21908 raft_consensus.cc:2454] T c0b87dc108f4403f8aa5dee538564479 P 83ce3c4dcf584675a8ee8c50c417e5fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a995b5225ea94d9ca0e29f51d924b8e4 in term 1.
I20260928 22:49:16.474390 21580 leader_election.cc:302] T c0b87dc108f4403f8aa5dee538564479 P a995b5225ea94d9ca0e29f51d924b8e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83ce3c4dcf584675a8ee8c50c417e5fa, a995b5225ea94d9ca0e29f51d924b8e4; no voters:
I20260928 22:49:16.474516 21976 raft_consensus.cc:2781] T c0b87dc108f4403f8aa5dee538564479 P a995b5225ea94d9ca0e29f51d924b8e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:49:16.474710 21976 raft_consensus.cc:686] T c0b87dc108f4403f8aa5dee538564479 P a995b5225ea94d9ca0e29f51d924b8e4 [term 1 LEADER]: Becoming Leader. State: Replica: a995b5225ea94d9ca0e29f51d924b8e4, State: Running, Role: LEADER
I20260928 22:49:16.474787 21976 consensus_queue.cc:237] T c0b87dc108f4403f8aa5dee538564479 P a995b5225ea94d9ca0e29f51d924b8e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ce3c4dcf584675a8ee8c50c417e5fa" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 38749 } } peers { permanent_uuid: "7e3833a70e3441148078ed102fcf7931" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32785 } } peers { permanent_uuid: "a995b5225ea94d9ca0e29f51d924b8e4" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33305 } }
I20260928 22:49:16.475751 21504 catalog_manager.cc:5495] T c0b87dc108f4403f8aa5dee538564479 P a995b5225ea94d9ca0e29f51d924b8e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to a995b5225ea94d9ca0e29f51d924b8e4 (127.16.121.129). New cstate: current_term: 1 leader_uuid: "a995b5225ea94d9ca0e29f51d924b8e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83ce3c4dcf584675a8ee8c50c417e5fa" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 38749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e3833a70e3441148078ed102fcf7931" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a995b5225ea94d9ca0e29f51d924b8e4" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33305 } health_report { overall_health: HEALTHY } } }
I20260928 22:49:16.522689 21967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:16.521513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43730 (local address 127.16.121.129:33305)
0928 22:49:16.521581 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:16.521583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:16.521592 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:49:16.521673 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:16.521674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:16.521692 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:16.521743 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:16.522435 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:16.522436 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:16.522436 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:49:16.522440 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:16.522457 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:49:16.522462 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:16.522525 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:16.522562 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 22:49:16.522575 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 22:49:16.524947 21777 raft_consensus.cc:1261] T c0b87dc108f4403f8aa5dee538564479 P 7e3833a70e3441148078ed102fcf7931 [term 1 FOLLOWER]: Refusing update from remote peer a995b5225ea94d9ca0e29f51d924b8e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:49:16.524950 21908 raft_consensus.cc:1261] T c0b87dc108f4403f8aa5dee538564479 P 83ce3c4dcf584675a8ee8c50c417e5fa [term 1 FOLLOWER]: Refusing update from remote peer a995b5225ea94d9ca0e29f51d924b8e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:49:16.525548 21978 consensus_queue.cc:1035] T c0b87dc108f4403f8aa5dee538564479 P a995b5225ea94d9ca0e29f51d924b8e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e3833a70e3441148078ed102fcf7931" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 32785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:49:16.525710 21976 consensus_queue.cc:1035] T c0b87dc108f4403f8aa5dee538564479 P a995b5225ea94d9ca0e29f51d924b8e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ce3c4dcf584675a8ee8c50c417e5fa" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 38749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:49:16.528925 21983 mvcc.cc:205] Tried to move back new op lower bound from 7334444058722267136 to 7334444058521210880. Current Snapshot: MvccSnapshot[applied={T|T < 7334444058722267136}]
I20260928 22:49:16.530855 21503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:49:16.530964 21503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:49:16.531858 21503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7ce941a5463f472eaca59401f0f93e97: Sending DeleteTablet for 3 replicas of tablet c0b87dc108f4403f8aa5dee538564479
I20260928 22:49:16.532236 21757 tablet_service.cc:1514] Processing DeleteTablet for tablet c0b87dc108f4403f8aa5dee538564479 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:49:16 UTC) from {username='slave'} at 127.0.0.1:41122
I20260928 22:49:16.532366 21888 tablet_service.cc:1514] Processing DeleteTablet for tablet c0b87dc108f4403f8aa5dee538564479 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:49:16 UTC) from {username='slave'} at 127.0.0.1:47538
I20260928 22:49:16.532512 21990 tablet_replica.cc:330] T c0b87dc108f4403f8aa5dee538564479 P 7e3833a70e3441148078ed102fcf7931: stopping tablet replica
I20260928 22:49:16.532488 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 21564
I20260928 22:49:16.532573 21626 tablet_service.cc:1514] Processing DeleteTablet for tablet c0b87dc108f4403f8aa5dee538564479 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:49:16 UTC) from {username='slave'} at 127.0.0.1:43726
I20260928 22:49:16.532666 21990 raft_consensus.cc:2229] T c0b87dc108f4403f8aa5dee538564479 P 7e3833a70e3441148078ed102fcf7931 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:49:16.532786 21990 raft_consensus.cc:2258] T c0b87dc108f4403f8aa5dee538564479 P 7e3833a70e3441148078ed102fcf7931 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:49:16.533361 21990 ts_tablet_manager.cc:1905] T c0b87dc108f4403f8aa5dee538564479 P 7e3833a70e3441148078ed102fcf7931: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:49:16.535507 21990 ts_tablet_manager.cc:1918] T c0b87dc108f4403f8aa5dee538564479 P 7e3833a70e3441148078ed102fcf7931: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:49:16.535600 21990 log.cc:1196] T c0b87dc108f4403f8aa5dee538564479 P 7e3833a70e3441148078ed102fcf7931: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-1/wal/wals/c0b87dc108f4403f8aa5dee538564479
I20260928 22:49:16.535869 21990 ts_tablet_manager.cc:1939] T c0b87dc108f4403f8aa5dee538564479 P 7e3833a70e3441148078ed102fcf7931: Deleting consensus metadata
I20260928 22:49:16.536689 21491 catalog_manager.cc:4841] TS 7e3833a70e3441148078ed102fcf7931 (127.16.121.130:32785): tablet c0b87dc108f4403f8aa5dee538564479 (table test-table [id=da8dc38055ee4fb7855136f663657bed]) successfully deleted
I20260928 22:49:16.539487 21991 tablet_replica.cc:330] T c0b87dc108f4403f8aa5dee538564479 P 83ce3c4dcf584675a8ee8c50c417e5fa: stopping tablet replica
W20260928 22:49:16.539554 21489 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.121.129:33305 (error 108)
I20260928 22:49:16.539664 21991 raft_consensus.cc:2229] T c0b87dc108f4403f8aa5dee538564479 P 83ce3c4dcf584675a8ee8c50c417e5fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:49:16.539676 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 21695
I20260928 22:49:16.539830 21991 raft_consensus.cc:2258] T c0b87dc108f4403f8aa5dee538564479 P 83ce3c4dcf584675a8ee8c50c417e5fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:49:16.541095 21991 ts_tablet_manager.cc:1905] T c0b87dc108f4403f8aa5dee538564479 P 83ce3c4dcf584675a8ee8c50c417e5fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:49:16.543258 21991 ts_tablet_manager.cc:1918] T c0b87dc108f4403f8aa5dee538564479 P 83ce3c4dcf584675a8ee8c50c417e5fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:49:16.543362 21991 log.cc:1196] T c0b87dc108f4403f8aa5dee538564479 P 83ce3c4dcf584675a8ee8c50c417e5fa: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790635733329745-16870-0/minicluster-data/ts-2/wal/wals/c0b87dc108f4403f8aa5dee538564479
I20260928 22:49:16.543665 21991 ts_tablet_manager.cc:1939] T c0b87dc108f4403f8aa5dee538564479 P 83ce3c4dcf584675a8ee8c50c417e5fa: Deleting consensus metadata
I20260928 22:49:16.544184 21489 catalog_manager.cc:4841] TS 83ce3c4dcf584675a8ee8c50c417e5fa (127.16.121.131:38749): tablet c0b87dc108f4403f8aa5dee538564479 (table test-table [id=da8dc38055ee4fb7855136f663657bed]) successfully deleted
I20260928 22:49:16.544827 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 21826
I20260928 22:49:16.546437 21963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:16.546260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:33305 (local address 127.0.0.1:43732)
0928 22:49:16.546332 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:16.546395 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.129:33305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 22:49:16.546656 21489 catalog_manager.cc:4584] TS a995b5225ea94d9ca0e29f51d924b8e4 (127.16.121.129:33305): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c0b87dc108f4403f8aa5dee538564479: Network error: Client connection negotiation failed: client connection to 127.16.121.129:33305: connect: Connection refused (error 111)
I20260928 22:49:16.549542 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 21473
2026-09-28T22:49:16Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (892 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 22:49:16.569526 16870 mini_oidc.cc:152] Starting JWKS server
I20260928 22:49:16.571239 16870 webserver.cc:460] Webserver started at https://127.0.0.1:45345/ using document root <none> and password file <none>
I20260928 22:49:16.571380 16870 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 22:49:16.571566 16870 webserver.cc:460] Webserver started at http://127.0.0.1:46153/ using document root <none> and password file <none>
2026-09-28T22:49:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:49:16Z Disabled control of system clock
I20260928 22:49:16.579813 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:39477
--webserver_interface=127.16.121.190
--webserver_port=0
--builtin_ntp_servers=127.16.121.148:34411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:39477
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45345/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790635733329745-16870-0/testchainca.cert with env {}
W20260928 22:49:16.667654 21998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:16.667882 21998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:16.667905 21998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:16.669153 21998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:49:16.669199 21998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:16.669212 21998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:16.669225 21998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:49:16.669237 21998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 22:49:16.669250 21998 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 22:49:16.669261 21998 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45345/jwks.json
I20260928 22:49:16.670885 21998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:34411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:39477
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:39477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45345/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.121.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790635733329745-16870-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:16.671262 21998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:16.671625 21998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:16.677018 22003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:16.677156 22006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:16.677306 22004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:16.677403 21998 server_base.cc:1034] running on GCE node
I20260928 22:49:16.677668 21998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:16.677953 21998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:16.679147 21998 hybrid_clock.cc:648] HybridClock initialized: now 1790635756679117 us; error 41 us; skew 500 ppm
I20260928 22:49:16.680361 21998 webserver.cc:460] Webserver started at http://127.16.121.190:45441/ using document root <none> and password file <none>
I20260928 22:49:16.680589 21998 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:16.680634 21998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:16.680724 21998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:16.681878 21998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "fb5cf4b6c0094d708a6ab9fc8b05335a"
format_stamp: "Formatted at 2026-09-28 22:49:16 on dist-test-slave-56vw"
server_key: "511c3bc1571f3ad84ad41a6a43236d1d"
server_key_iv: "34b7423a9d57beca8e12d5a962106de6"
server_key_version: "encryptionkey@0"
I20260928 22:49:16.682200 21998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "fb5cf4b6c0094d708a6ab9fc8b05335a"
format_stamp: "Formatted at 2026-09-28 22:49:16 on dist-test-slave-56vw"
server_key: "511c3bc1571f3ad84ad41a6a43236d1d"
server_key_iv: "34b7423a9d57beca8e12d5a962106de6"
server_key_version: "encryptionkey@0"
I20260928 22:49:16.683573 21998 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:49:16.684362 22012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:16.684608 21998 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:16.684721 21998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "fb5cf4b6c0094d708a6ab9fc8b05335a"
format_stamp: "Formatted at 2026-09-28 22:49:16 on dist-test-slave-56vw"
server_key: "511c3bc1571f3ad84ad41a6a43236d1d"
server_key_iv: "34b7423a9d57beca8e12d5a962106de6"
server_key_version: "encryptionkey@0"
I20260928 22:49:16.684803 21998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:16.696849 21998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:16.697337 21998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:16.697508 21998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:16.701902 21998 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.190:39477
I20260928 22:49:16.701947 22064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.190:39477 every 8 connection(s)
I20260928 22:49:16.702349 21998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:49:16.703469 22065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:16.704867 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 21998
I20260928 22:49:16.705004 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:49:16.705196 16870 external_mini_cluster.cc:1428] Setting key 7b3611eb7d3510f260fe304069094737
I20260928 22:49:16.706351 22065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb5cf4b6c0094d708a6ab9fc8b05335a: Bootstrap starting.
I20260928 22:49:16.707149 22065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb5cf4b6c0094d708a6ab9fc8b05335a: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:16.707465 22065 log.cc:824] T 00000000000000000000000000000000 P fb5cf4b6c0094d708a6ab9fc8b05335a: Log is configured to *not* fsync() on all Append() calls
I20260928 22:49:16.708333 22065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb5cf4b6c0094d708a6ab9fc8b05335a: No bootstrap required, opened a new log
I20260928 22:49:16.710742 22065 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb5cf4b6c0094d708a6ab9fc8b05335a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb5cf4b6c0094d708a6ab9fc8b05335a" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39477 } }
I20260928 22:49:16.710930 22065 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb5cf4b6c0094d708a6ab9fc8b05335a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:16.710984 22065 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb5cf4b6c0094d708a6ab9fc8b05335a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb5cf4b6c0094d708a6ab9fc8b05335a, State: Initialized, Role: FOLLOWER
I20260928 22:49:16.711189 22065 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb5cf4b6c0094d708a6ab9fc8b05335a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb5cf4b6c0094d708a6ab9fc8b05335a" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39477 } }
I20260928 22:49:16.711283 22065 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb5cf4b6c0094d708a6ab9fc8b05335a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:49:16.711308 22065 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb5cf4b6c0094d708a6ab9fc8b05335a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:49:16.711380 22065 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb5cf4b6c0094d708a6ab9fc8b05335a [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:16.711447 22068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:16.705747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44174 (local address 127.16.121.190:39477)
0928 22:49:16.705974 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:16.705981 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:16.706007 (+ 26us) server_negotiation.cc:407] Connection header received
0928 22:49:16.706427 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:16.706436 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:16.706525 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:16.706659 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:16.706919 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:16.708124 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:16.709275 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:16.709739 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:16.709840 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:16.710588 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:16.710592 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:16.710592 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:49:16.710763 (+ 171us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:16.710798 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:49:16.710809 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:16.710862 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:16.711042 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 22:49:16.711182 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":95,"threads_started":1}
I20260928 22:49:16.712133 22065 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb5cf4b6c0094d708a6ab9fc8b05335a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb5cf4b6c0094d708a6ab9fc8b05335a" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39477 } }
I20260928 22:49:16.712266 22065 leader_election.cc:302] T 00000000000000000000000000000000 P fb5cf4b6c0094d708a6ab9fc8b05335a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb5cf4b6c0094d708a6ab9fc8b05335a; no voters:
I20260928 22:49:16.712457 22065 leader_election.cc:288] T 00000000000000000000000000000000 P fb5cf4b6c0094d708a6ab9fc8b05335a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:49:16.712556 22070 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb5cf4b6c0094d708a6ab9fc8b05335a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:49:16.712584 16870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:49:16.712771 22070 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb5cf4b6c0094d708a6ab9fc8b05335a [term 1 LEADER]: Becoming Leader. State: Replica: fb5cf4b6c0094d708a6ab9fc8b05335a, State: Running, Role: LEADER
I20260928 22:49:16.712800 22065 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb5cf4b6c0094d708a6ab9fc8b05335a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:49:16.712857 22070 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb5cf4b6c0094d708a6ab9fc8b05335a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb5cf4b6c0094d708a6ab9fc8b05335a" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39477 } }
I20260928 22:49:16.713186 22071 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb5cf4b6c0094d708a6ab9fc8b05335a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb5cf4b6c0094d708a6ab9fc8b05335a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb5cf4b6c0094d708a6ab9fc8b05335a" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39477 } } }
I20260928 22:49:16.713274 22071 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb5cf4b6c0094d708a6ab9fc8b05335a [sys.catalog]: This master's current role is: LEADER
I20260928 22:49:16.713171 22070 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb5cf4b6c0094d708a6ab9fc8b05335a [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb5cf4b6c0094d708a6ab9fc8b05335a. Latest consensus state: current_term: 1 leader_uuid: "fb5cf4b6c0094d708a6ab9fc8b05335a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb5cf4b6c0094d708a6ab9fc8b05335a" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39477 } } }
I20260928 22:49:16.713486 22070 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb5cf4b6c0094d708a6ab9fc8b05335a [sys.catalog]: This master's current role is: LEADER
W20260928 22:49:16.714548 22086 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fb5cf4b6c0094d708a6ab9fc8b05335a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:49:16.714593 22086 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:49:16.714654 22077 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:49:16.714815 22077 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:49:16.716298 22077 catalog_manager.cc:1347] Generated new cluster ID: 62bb410d3692464186edd0a409a3bdb2
I20260928 22:49:16.716354 22077 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:49:16.735277 22077 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:49:16.735881 22077 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:49:16.740674 22077 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb5cf4b6c0094d708a6ab9fc8b05335a: Generated new TSK 0
I20260928 22:49:16.740917 22077 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 22:49:16.770196 22095 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 22:49:16.770323 22095 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260928 22:49:16.770843 22094 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:16.754940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:39477 (local address 127.0.0.1:44182)
0928 22:49:16.755195 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:16.755206 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:16.755303 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:16.755519 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:16.755520 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:16.755523 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0928 22:49:16.755608 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:16.755611 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:16.756324 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:16.756326 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:16.756944 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:16.756946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:16.757353 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:16.757354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:16.757423 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:16.757461 (+ 38us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 22:49:16.770455 (+ 12994us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45345/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 22:49:16.770582 (+ 127us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.121.190:39477: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45345/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":89,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.16.121.190:39477: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45345/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 22:49:16.771265 22068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:16.755039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44182 (local address 127.16.121.190:39477)
0928 22:49:16.755105 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:16.755107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:16.755277 (+ 170us) server_negotiation.cc:407] Connection header received
0928 22:49:16.755400 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:16.755400 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:16.755412 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:16.755495 (+ 83us) server_negotiation.cc:226] Negotiated authn=JWT
0928 22:49:16.755688 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:16.756197 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:16.757057 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:16.757269 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:16.757351 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:16.757604 (+ 253us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 22:49:16.770289 (+ 12685us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45345/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 22:49:16.771197 (+ 908us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44182: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45345/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":38}
W20260928 22:49:16.771313 22068 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44182: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45345/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
I20260928 22:49:16.771306 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 21998
2026-09-28T22:49:16Z chronyd exiting
I20260928 22:49:16.807837 16870 test_util.cc:175] -----------------------------------------------
I20260928 22:49:16.807932 16870 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790635733329745-16870-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (239 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 22:49:16.808810 16870 mini_oidc.cc:152] Starting JWKS server
I20260928 22:49:16.809479 16870 webserver.cc:460] Webserver started at https://127.0.0.1:43807/ using document root <none> and password file <none>
I20260928 22:49:16.809517 16870 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 22:49:16.809659 16870 webserver.cc:460] Webserver started at http://127.0.0.1:37433/ using document root <none> and password file <none>
2026-09-28T22:49:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:49:16Z Disabled control of system clock
I20260928 22:49:16.819300 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:38481
--webserver_interface=127.16.121.190
--webserver_port=0
--builtin_ntp_servers=127.16.121.148:35109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:38481
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43807/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790635733329745-16870-0/testchainca.cert with env {}
W20260928 22:49:16.904408 22101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:16.904686 22101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:16.904731 22101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:16.906003 22101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:49:16.906078 22101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:16.906124 22101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:16.906155 22101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:49:16.906184 22101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 22:49:16.906213 22101 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 22:49:16.906252 22101 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43807/jwks.json
I20260928 22:49:16.907969 22101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:35109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:38481
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:38481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43807/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.121.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790635733329745-16870-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:16.908284 22101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:16.908525 22101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:16.914201 22109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:16.914201 22107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:16.914273 22106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:16.914537 22101 server_base.cc:1034] running on GCE node
I20260928 22:49:16.914860 22101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:16.915282 22101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:16.916671 22101 hybrid_clock.cc:648] HybridClock initialized: now 1790635756916654 us; error 35 us; skew 500 ppm
I20260928 22:49:16.918262 22101 webserver.cc:460] Webserver started at http://127.16.121.190:44969/ using document root <none> and password file <none>
I20260928 22:49:16.918653 22101 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:16.918733 22101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:16.918897 22101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:16.920086 22101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "b79527adaa8247bcb1ee8ea1eb8af9f9"
format_stamp: "Formatted at 2026-09-28 22:49:16 on dist-test-slave-56vw"
server_key: "aeb9b49f1387e353a9466463ebcf0e7a"
server_key_iv: "6ab72621a104cc99540d27fe6db30db3"
server_key_version: "encryptionkey@0"
I20260928 22:49:16.920477 22101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "b79527adaa8247bcb1ee8ea1eb8af9f9"
format_stamp: "Formatted at 2026-09-28 22:49:16 on dist-test-slave-56vw"
server_key: "aeb9b49f1387e353a9466463ebcf0e7a"
server_key_iv: "6ab72621a104cc99540d27fe6db30db3"
server_key_version: "encryptionkey@0"
I20260928 22:49:16.921867 22101 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:49:16.922875 22115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:16.923118 22101 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:16.923220 22101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "b79527adaa8247bcb1ee8ea1eb8af9f9"
format_stamp: "Formatted at 2026-09-28 22:49:16 on dist-test-slave-56vw"
server_key: "aeb9b49f1387e353a9466463ebcf0e7a"
server_key_iv: "6ab72621a104cc99540d27fe6db30db3"
server_key_version: "encryptionkey@0"
I20260928 22:49:16.923393 22101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:16.932032 22101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:16.932387 22101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:16.932559 22101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:16.937199 22101 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.190:38481
I20260928 22:49:16.937230 22167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.190:38481 every 8 connection(s)
I20260928 22:49:16.937628 22101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:49:16.938393 22168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:16.941352 22168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b79527adaa8247bcb1ee8ea1eb8af9f9: Bootstrap starting.
I20260928 22:49:16.941943 22168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b79527adaa8247bcb1ee8ea1eb8af9f9: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:16.942221 22168 log.cc:824] T 00000000000000000000000000000000 P b79527adaa8247bcb1ee8ea1eb8af9f9: Log is configured to *not* fsync() on all Append() calls
I20260928 22:49:16.943092 22168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b79527adaa8247bcb1ee8ea1eb8af9f9: No bootstrap required, opened a new log
I20260928 22:49:16.944116 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 22101
I20260928 22:49:16.944247 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:49:16.944440 16870 external_mini_cluster.cc:1428] Setting key 84939eb539adc979836c4e49c1e52450
I20260928 22:49:16.945261 22168 raft_consensus.cc:348] T 00000000000000000000000000000000 P b79527adaa8247bcb1ee8ea1eb8af9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b79527adaa8247bcb1ee8ea1eb8af9f9" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 38481 } }
I20260928 22:49:16.945430 22168 raft_consensus.cc:374] T 00000000000000000000000000000000 P b79527adaa8247bcb1ee8ea1eb8af9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:16.945475 22168 raft_consensus.cc:729] T 00000000000000000000000000000000 P b79527adaa8247bcb1ee8ea1eb8af9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b79527adaa8247bcb1ee8ea1eb8af9f9, State: Initialized, Role: FOLLOWER
I20260928 22:49:16.945619 22168 consensus_queue.cc:260] T 00000000000000000000000000000000 P b79527adaa8247bcb1ee8ea1eb8af9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b79527adaa8247bcb1ee8ea1eb8af9f9" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 38481 } }
I20260928 22:49:16.945711 22168 raft_consensus.cc:388] T 00000000000000000000000000000000 P b79527adaa8247bcb1ee8ea1eb8af9f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:49:16.945751 22168 raft_consensus.cc:482] T 00000000000000000000000000000000 P b79527adaa8247bcb1ee8ea1eb8af9f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:49:16.945815 22168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b79527adaa8247bcb1ee8ea1eb8af9f9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:16.946384 22168 raft_consensus.cc:504] T 00000000000000000000000000000000 P b79527adaa8247bcb1ee8ea1eb8af9f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b79527adaa8247bcb1ee8ea1eb8af9f9" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 38481 } }
I20260928 22:49:16.946504 22168 leader_election.cc:302] T 00000000000000000000000000000000 P b79527adaa8247bcb1ee8ea1eb8af9f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b79527adaa8247bcb1ee8ea1eb8af9f9; no voters:
I20260928 22:49:16.946687 22168 leader_election.cc:288] T 00000000000000000000000000000000 P b79527adaa8247bcb1ee8ea1eb8af9f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:49:16.946735 22173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b79527adaa8247bcb1ee8ea1eb8af9f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:49:16.946877 22173 raft_consensus.cc:686] T 00000000000000000000000000000000 P b79527adaa8247bcb1ee8ea1eb8af9f9 [term 1 LEADER]: Becoming Leader. State: Replica: b79527adaa8247bcb1ee8ea1eb8af9f9, State: Running, Role: LEADER
I20260928 22:49:16.946972 22168 sys_catalog.cc:564] T 00000000000000000000000000000000 P b79527adaa8247bcb1ee8ea1eb8af9f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:49:16.946964 22173 consensus_queue.cc:237] T 00000000000000000000000000000000 P b79527adaa8247bcb1ee8ea1eb8af9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b79527adaa8247bcb1ee8ea1eb8af9f9" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 38481 } }
I20260928 22:49:16.947525 22174 sys_catalog.cc:455] T 00000000000000000000000000000000 P b79527adaa8247bcb1ee8ea1eb8af9f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b79527adaa8247bcb1ee8ea1eb8af9f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b79527adaa8247bcb1ee8ea1eb8af9f9" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 38481 } } }
I20260928 22:49:16.947538 22175 sys_catalog.cc:455] T 00000000000000000000000000000000 P b79527adaa8247bcb1ee8ea1eb8af9f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b79527adaa8247bcb1ee8ea1eb8af9f9. Latest consensus state: current_term: 1 leader_uuid: "b79527adaa8247bcb1ee8ea1eb8af9f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b79527adaa8247bcb1ee8ea1eb8af9f9" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 38481 } } }
I20260928 22:49:16.947654 22174 sys_catalog.cc:458] T 00000000000000000000000000000000 P b79527adaa8247bcb1ee8ea1eb8af9f9 [sys.catalog]: This master's current role is: LEADER
I20260928 22:49:16.947659 22175 sys_catalog.cc:458] T 00000000000000000000000000000000 P b79527adaa8247bcb1ee8ea1eb8af9f9 [sys.catalog]: This master's current role is: LEADER
I20260928 22:49:16.948307 22178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:49:16.948860 22178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:49:16.950618 22178 catalog_manager.cc:1347] Generated new cluster ID: 90fd2297bd454fb684de8691926ed8b4
I20260928 22:49:16.950675 22178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:49:16.951090 22172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:16.944988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48272 (local address 127.16.121.190:38481)
0928 22:49:16.945204 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:16.945212 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:16.945239 (+ 27us) server_negotiation.cc:407] Connection header received
0928 22:49:16.945335 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:16.945340 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:16.945399 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:16.945554 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:16.945778 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:16.947472 (+ 1694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:16.948714 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:16.949321 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:16.949449 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:16.950293 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:16.950297 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:16.950298 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:49:16.950475 (+ 177us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:16.950543 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:49:16.950560 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:16.950633 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:16.950828 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 22:49:16.950891 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":100,"threads_started":1}
I20260928 22:49:16.956490 16870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:49:16.977293 22178 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:49:16.977916 22178 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:49:16.982625 22178 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b79527adaa8247bcb1ee8ea1eb8af9f9: Generated new TSK 0
I20260928 22:49:16.982890 22178 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 22:49:17.007519 22198 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 22:49:17.007616 22198 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 22:49:17.007997 22172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:16.992715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48280 (local address 127.16.121.190:38481)
0928 22:49:16.992785 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:16.992788 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:16.992888 (+ 100us) server_negotiation.cc:407] Connection header received
0928 22:49:16.992968 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:16.992969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:16.992980 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:16.993044 (+ 64us) server_negotiation.cc:226] Negotiated authn=JWT
0928 22:49:16.993187 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:16.993694 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:16.994841 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:16.995113 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:16.995178 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:16.995528 (+ 350us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 22:49:17.007783 (+ 12255us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43807/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 22:49:17.007932 (+ 149us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48280: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43807/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":41}
W20260928 22:49:17.008069 22172 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48280: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43807/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 22:49:17.008172 22197 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:16.992654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:38481 (local address 127.0.0.1:48280)
0928 22:49:16.992810 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:16.992822 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:16.992895 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:16.993063 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:16.993064 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:16.993066 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 22:49:16.993123 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:16.993126 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:16.993775 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:16.993776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:16.994736 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:16.994739 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:16.995195 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:16.995196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:16.995268 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:16.995275 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 22:49:17.007920 (+ 12645us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43807/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 22:49:17.008068 (+ 148us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.121.190:38481: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43807/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":105,"threads_started":1}
I20260928 22:49:17.008494 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 22101
2026-09-28T22:49:17Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (237 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 22:49:17.046415 16870 mini_oidc.cc:152] Starting JWKS server
I20260928 22:49:17.047223 16870 webserver.cc:460] Webserver started at https://127.0.0.1:40937/ using document root <none> and password file <none>
I20260928 22:49:17.047299 16870 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 22:49:17.047459 16870 webserver.cc:460] Webserver started at http://127.0.0.1:45275/ using document root <none> and password file <none>
2026-09-28T22:49:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:49:17Z Disabled control of system clock
I20260928 22:49:17.056169 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:41955
--webserver_interface=127.16.121.190
--webserver_port=0
--builtin_ntp_servers=127.16.121.148:45731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:41955
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40937/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 22:49:17.142326 22204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:17.142575 22204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:17.142599 22204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:17.143953 22204 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:49:17.144013 22204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:17.144026 22204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:17.144040 22204 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:49:17.144052 22204 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 22:49:17.144065 22204 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 22:49:17.144076 22204 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40937/jwks.json
I20260928 22:49:17.145637 22204 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:45731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:41955
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:41955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40937/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.121.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:17.145931 22204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:17.146268 22204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:17.151542 22209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:17.151593 22210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:17.151582 22212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:17.151638 22204 server_base.cc:1034] running on GCE node
I20260928 22:49:17.151974 22204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:17.152254 22204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:17.153415 22204 hybrid_clock.cc:648] HybridClock initialized: now 1790635757153393 us; error 42 us; skew 500 ppm
I20260928 22:49:17.155027 22204 webserver.cc:460] Webserver started at http://127.16.121.190:37195/ using document root <none> and password file <none>
I20260928 22:49:17.155391 22204 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:17.155471 22204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:17.155616 22204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:17.156736 22204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "5df0cffd1adb4410b45d57d491491d87"
format_stamp: "Formatted at 2026-09-28 22:49:17 on dist-test-slave-56vw"
server_key: "05fc202360d3c7d0486ee07cba873fb0"
server_key_iv: "20118ec28cb4b3431a5a59a23f46806c"
server_key_version: "encryptionkey@0"
I20260928 22:49:17.157131 22204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "5df0cffd1adb4410b45d57d491491d87"
format_stamp: "Formatted at 2026-09-28 22:49:17 on dist-test-slave-56vw"
server_key: "05fc202360d3c7d0486ee07cba873fb0"
server_key_iv: "20118ec28cb4b3431a5a59a23f46806c"
server_key_version: "encryptionkey@0"
I20260928 22:49:17.158602 22204 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:49:17.159734 22218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:17.159956 22204 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 22:49:17.160079 22204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "5df0cffd1adb4410b45d57d491491d87"
format_stamp: "Formatted at 2026-09-28 22:49:17 on dist-test-slave-56vw"
server_key: "05fc202360d3c7d0486ee07cba873fb0"
server_key_iv: "20118ec28cb4b3431a5a59a23f46806c"
server_key_version: "encryptionkey@0"
I20260928 22:49:17.160163 22204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:17.185575 22204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:17.185948 22204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:17.186115 22204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:17.190456 22204 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.190:41955
I20260928 22:49:17.190502 22270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.190:41955 every 8 connection(s)
I20260928 22:49:17.190886 22204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:49:17.191201 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 22204
I20260928 22:49:17.191314 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:49:17.191545 16870 external_mini_cluster.cc:1428] Setting key 2fd60a094af9edfa6244ca5690ad159a
I20260928 22:49:17.191725 22271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:17.195075 22271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5df0cffd1adb4410b45d57d491491d87: Bootstrap starting.
I20260928 22:49:17.195681 22271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5df0cffd1adb4410b45d57d491491d87: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:17.196008 22271 log.cc:824] T 00000000000000000000000000000000 P 5df0cffd1adb4410b45d57d491491d87: Log is configured to *not* fsync() on all Append() calls
I20260928 22:49:17.196792 22271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5df0cffd1adb4410b45d57d491491d87: No bootstrap required, opened a new log
I20260928 22:49:17.198254 22274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:17.192457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57122 (local address 127.16.121.190:41955)
0928 22:49:17.192655 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:17.192665 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:17.192697 (+ 32us) server_negotiation.cc:407] Connection header received
0928 22:49:17.193211 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:17.193220 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:17.193309 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:17.193474 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:17.193679 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:17.195081 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:17.195978 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:17.196549 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:17.196677 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:17.197460 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:17.197466 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:17.197466 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:49:17.197676 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:17.197721 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:49:17.197736 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:17.197809 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:17.197949 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 22:49:17.198040 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":61,"threads_started":1}
I20260928 22:49:17.199299 16870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:49:17.199447 22271 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5df0cffd1adb4410b45d57d491491d87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5df0cffd1adb4410b45d57d491491d87" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 41955 } }
I20260928 22:49:17.199627 22271 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5df0cffd1adb4410b45d57d491491d87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:17.199652 22271 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5df0cffd1adb4410b45d57d491491d87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5df0cffd1adb4410b45d57d491491d87, State: Initialized, Role: FOLLOWER
I20260928 22:49:17.199791 22271 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5df0cffd1adb4410b45d57d491491d87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5df0cffd1adb4410b45d57d491491d87" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 41955 } }
I20260928 22:49:17.199873 22271 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5df0cffd1adb4410b45d57d491491d87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:49:17.199937 22271 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5df0cffd1adb4410b45d57d491491d87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:49:17.199990 22271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5df0cffd1adb4410b45d57d491491d87 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:17.200812 22271 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5df0cffd1adb4410b45d57d491491d87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5df0cffd1adb4410b45d57d491491d87" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 41955 } }
I20260928 22:49:17.200938 22271 leader_election.cc:302] T 00000000000000000000000000000000 P 5df0cffd1adb4410b45d57d491491d87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5df0cffd1adb4410b45d57d491491d87; no voters:
I20260928 22:49:17.201138 22271 leader_election.cc:288] T 00000000000000000000000000000000 P 5df0cffd1adb4410b45d57d491491d87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:49:17.201177 22276 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5df0cffd1adb4410b45d57d491491d87 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:49:17.201327 22276 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5df0cffd1adb4410b45d57d491491d87 [term 1 LEADER]: Becoming Leader. State: Replica: 5df0cffd1adb4410b45d57d491491d87, State: Running, Role: LEADER
I20260928 22:49:17.201406 22271 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5df0cffd1adb4410b45d57d491491d87 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:49:17.201435 22276 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5df0cffd1adb4410b45d57d491491d87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5df0cffd1adb4410b45d57d491491d87" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 41955 } }
I20260928 22:49:17.201891 22277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5df0cffd1adb4410b45d57d491491d87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5df0cffd1adb4410b45d57d491491d87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5df0cffd1adb4410b45d57d491491d87" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 41955 } } }
I20260928 22:49:17.201975 22277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5df0cffd1adb4410b45d57d491491d87 [sys.catalog]: This master's current role is: LEADER
I20260928 22:49:17.202160 22278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5df0cffd1adb4410b45d57d491491d87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5df0cffd1adb4410b45d57d491491d87. Latest consensus state: current_term: 1 leader_uuid: "5df0cffd1adb4410b45d57d491491d87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5df0cffd1adb4410b45d57d491491d87" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 41955 } } }
I20260928 22:49:17.202198 22284 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:49:17.202227 22278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5df0cffd1adb4410b45d57d491491d87 [sys.catalog]: This master's current role is: LEADER
I20260928 22:49:17.202836 22284 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:49:17.204577 22284 catalog_manager.cc:1347] Generated new cluster ID: b91232b751794d0c820a207960528779
I20260928 22:49:17.204634 22284 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:49:17.225416 22284 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:49:17.226096 22284 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:49:17.231340 22284 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5df0cffd1adb4410b45d57d491491d87: Generated new TSK 0
I20260928 22:49:17.231566 22284 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:49:17.242791 22274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:17.238378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57132 (local address 127.16.121.190:41955)
0928 22:49:17.238428 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:17.238430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:17.238543 (+ 113us) server_negotiation.cc:407] Connection header received
0928 22:49:17.238598 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:17.238599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:17.238611 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:17.238680 (+ 69us) server_negotiation.cc:226] Negotiated authn=JWT
0928 22:49:17.238844 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:17.239412 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:17.240589 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:17.240784 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:17.240841 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:17.241075 (+ 234us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 22:49:17.242534 (+ 1459us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40937/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 22:49:17.242709 (+ 175us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57132: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40937/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":26}
W20260928 22:49:17.242864 22274 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57132: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40937/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 22:49:17.242867 22300 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:17.238262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:41955 (local address 127.0.0.1:57132)
0928 22:49:17.238440 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:17.238453 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:17.238528 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:17.238698 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:17.238698 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:17.238701 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0928 22:49:17.238774 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:17.238777 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:17.239515 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:17.239516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:17.240495 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:17.240499 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:17.240875 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:17.240876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:17.240940 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:17.240946 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 22:49:17.242654 (+ 1708us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40937/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 22:49:17.242757 (+ 103us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.121.190:41955: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40937/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":106,"threads_started":1}
I20260928 22:49:17.243496 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 22204
E20260928 22:49:17.255618 22301 webserver.cc:533] Webserver: sslize failed:
E20260928 22:49:17.255715 22301 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T22:49:17Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (236 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T22:49:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:49:17Z Disabled control of system clock
I20260928 22:49:17.292882 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:41733
--webserver_interface=127.16.121.190
--webserver_port=0
--builtin_ntp_servers=127.16.121.148:37351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:41733
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790635733329745-16870-0/testchainca.cert with env {}
W20260928 22:49:17.384661 22305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:17.384887 22305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:17.384910 22305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:17.386071 22305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:49:17.386117 22305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:17.386130 22305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:17.386143 22305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:49:17.386155 22305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 22:49:17.386168 22305 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 22:49:17.386179 22305 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 22:49:17.387738 22305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:37351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:41733
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:41733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.121.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790635733329745-16870-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:17.387974 22305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:17.388211 22305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:17.393379 22310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:17.393474 22305 server_base.cc:1034] running on GCE node
W20260928 22:49:17.393389 22311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:17.393389 22313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:17.393888 22305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:17.394214 22305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:17.395432 22305 hybrid_clock.cc:648] HybridClock initialized: now 1790635757395426 us; error 39 us; skew 500 ppm
I20260928 22:49:17.397039 22305 webserver.cc:460] Webserver started at http://127.16.121.190:39363/ using document root <none> and password file <none>
I20260928 22:49:17.397341 22305 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:17.397435 22305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:17.397603 22305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:17.398914 22305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "63ac66deaee64121b85ed0328fb34184"
format_stamp: "Formatted at 2026-09-28 22:49:17 on dist-test-slave-56vw"
server_key: "94ea440263b3dd5940c792fd6f5863df"
server_key_iv: "3fa5516e357a44c71f08c392929d1a16"
server_key_version: "encryptionkey@0"
I20260928 22:49:17.399519 22305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "63ac66deaee64121b85ed0328fb34184"
format_stamp: "Formatted at 2026-09-28 22:49:17 on dist-test-slave-56vw"
server_key: "94ea440263b3dd5940c792fd6f5863df"
server_key_iv: "3fa5516e357a44c71f08c392929d1a16"
server_key_version: "encryptionkey@0"
I20260928 22:49:17.401170 22305 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:49:17.402283 22319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:17.402697 22305 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:17.402832 22305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "63ac66deaee64121b85ed0328fb34184"
format_stamp: "Formatted at 2026-09-28 22:49:17 on dist-test-slave-56vw"
server_key: "94ea440263b3dd5940c792fd6f5863df"
server_key_iv: "3fa5516e357a44c71f08c392929d1a16"
server_key_version: "encryptionkey@0"
I20260928 22:49:17.402977 22305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:17.416102 22305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:17.416452 22305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:17.416610 22305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:17.420749 22305 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.190:41733
I20260928 22:49:17.420787 22371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.190:41733 every 8 connection(s)
I20260928 22:49:17.421164 22305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:49:17.422207 22372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:17.424594 22372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63ac66deaee64121b85ed0328fb34184: Bootstrap starting.
I20260928 22:49:17.425137 22372 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63ac66deaee64121b85ed0328fb34184: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:17.425374 22372 log.cc:824] T 00000000000000000000000000000000 P 63ac66deaee64121b85ed0328fb34184: Log is configured to *not* fsync() on all Append() calls
I20260928 22:49:17.426034 22372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63ac66deaee64121b85ed0328fb34184: No bootstrap required, opened a new log
I20260928 22:49:17.427721 22372 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63ac66deaee64121b85ed0328fb34184 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ac66deaee64121b85ed0328fb34184" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 41733 } }
I20260928 22:49:17.427861 22372 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63ac66deaee64121b85ed0328fb34184 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:17.427882 22372 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63ac66deaee64121b85ed0328fb34184 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63ac66deaee64121b85ed0328fb34184, State: Initialized, Role: FOLLOWER
I20260928 22:49:17.428020 22372 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63ac66deaee64121b85ed0328fb34184 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ac66deaee64121b85ed0328fb34184" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 41733 } }
I20260928 22:49:17.428099 22372 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63ac66deaee64121b85ed0328fb34184 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:49:17.428123 22372 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63ac66deaee64121b85ed0328fb34184 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:49:17.428151 22372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63ac66deaee64121b85ed0328fb34184 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:17.428663 22372 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63ac66deaee64121b85ed0328fb34184 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ac66deaee64121b85ed0328fb34184" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 41733 } }
I20260928 22:49:17.428777 22372 leader_election.cc:302] T 00000000000000000000000000000000 P 63ac66deaee64121b85ed0328fb34184 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63ac66deaee64121b85ed0328fb34184; no voters:
I20260928 22:49:17.428838 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 22305
I20260928 22:49:17.428939 22372 leader_election.cc:288] T 00000000000000000000000000000000 P 63ac66deaee64121b85ed0328fb34184 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:49:17.428957 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:49:17.429024 22375 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63ac66deaee64121b85ed0328fb34184 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:49:17.429155 16870 external_mini_cluster.cc:1428] Setting key bec06e284999f7736aedb8d7457249f5
I20260928 22:49:17.429189 22375 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63ac66deaee64121b85ed0328fb34184 [term 1 LEADER]: Becoming Leader. State: Replica: 63ac66deaee64121b85ed0328fb34184, State: Running, Role: LEADER
I20260928 22:49:17.429210 22372 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63ac66deaee64121b85ed0328fb34184 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:49:17.429308 22375 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63ac66deaee64121b85ed0328fb34184 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ac66deaee64121b85ed0328fb34184" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 41733 } }
I20260928 22:49:17.429666 22375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63ac66deaee64121b85ed0328fb34184 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63ac66deaee64121b85ed0328fb34184. Latest consensus state: current_term: 1 leader_uuid: "63ac66deaee64121b85ed0328fb34184" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ac66deaee64121b85ed0328fb34184" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 41733 } } }
I20260928 22:49:17.429755 22375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63ac66deaee64121b85ed0328fb34184 [sys.catalog]: This master's current role is: LEADER
I20260928 22:49:17.429878 22376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63ac66deaee64121b85ed0328fb34184 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63ac66deaee64121b85ed0328fb34184" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ac66deaee64121b85ed0328fb34184" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 41733 } } }
I20260928 22:49:17.430001 22376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63ac66deaee64121b85ed0328fb34184 [sys.catalog]: This master's current role is: LEADER
I20260928 22:49:17.430115 22384 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:49:17.430647 22384 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:49:17.432399 22384 catalog_manager.cc:1347] Generated new cluster ID: b60268fc9d304c3a8abdc85fabf6deda
I20260928 22:49:17.432456 22384 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:49:17.435204 22385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:17.429801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48842 (local address 127.16.121.190:41733)
0928 22:49:17.429996 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:17.430006 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:17.430277 (+ 271us) server_negotiation.cc:407] Connection header received
0928 22:49:17.430423 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:17.430431 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:17.430519 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:17.430697 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:17.430949 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:17.432341 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:17.433206 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:17.433634 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:17.433762 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:17.434481 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:17.434485 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:17.434486 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:49:17.434667 (+ 181us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:17.434699 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:49:17.434711 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:17.434782 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:17.434857 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 22:49:17.434940 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":75,"threads_started":1}
I20260928 22:49:17.436141 16870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:49:17.438884 22384 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:49:17.439414 22384 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:49:17.451040 22384 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63ac66deaee64121b85ed0328fb34184: Generated new TSK 0
I20260928 22:49:17.451318 22384 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:49:17.937863 22385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:17.461658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48848 (local address 127.16.121.190:41733)
0928 22:49:17.461734 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:17.461736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:17.461793 (+ 57us) server_negotiation.cc:407] Connection header received
0928 22:49:17.461874 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:17.461875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:17.461889 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:17.461967 (+ 78us) server_negotiation.cc:226] Negotiated authn=JWT
0928 22:49:17.462135 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:17.463181 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:17.464148 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:17.464511 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:17.464597 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:17.464828 (+ 231us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 22:49:17.937614 (+472786us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 22:49:17.937795 (+ 181us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48848: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":47}
W20260928 22:49:17.937929 22385 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48848: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 22:49:17.938043 22401 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:17.461397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:41733 (local address 127.0.0.1:48848)
0928 22:49:17.461715 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:17.461725 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:17.461793 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:17.461997 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:17.461998 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:17.462000 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 22:49:17.462060 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:17.462063 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:17.463342 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:17.463344 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:17.464041 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:17.464045 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:17.464641 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:17.464642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:17.464696 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:17.464701 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 22:49:17.937771 (+473070us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 22:49:17.937900 (+ 129us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.121.190:41733: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":220,"threads_started":1}
I20260928 22:49:17.938383 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 22305
2026-09-28T22:49:17Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (673 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (92 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (95 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:49:18 dist-test-slave-56vw krb5kdc[22413](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:49:18 dist-test-slave-56vw krb5kdc[22413](info): set up 2 sockets
Sep 28 22:49:18 dist-test-slave-56vw krb5kdc[22413](info): commencing operation
krb5kdc: starting...
W20260928 22:49:19.203626 16870 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Sep 28 22:49:19 dist-test-slave-56vw krb5kdc[22413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:49:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:49:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.121.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
I20260928 22:49:19.349119 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:37045
--webserver_interface=127.16.121.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:34587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:37045
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:49:19.441891 22426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:19.442211 22426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:19.442276 22426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:19.443605 22426 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:49:19.443696 22426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:19.443722 22426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:19.443742 22426 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:49:19.443771 22426 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:49:19.445462 22426 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:34587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:37045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:37045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.121.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:19.445798 22426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:19.446050 22426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:19.451720 22431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:19.451713 22432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:19.451713 22434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:19.451740 22426 server_base.cc:1034] running on GCE node
I20260928 22:49:19.452200 22426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:19.452541 22426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:19.453755 22426 hybrid_clock.cc:648] HybridClock initialized: now 1790635759453732 us; error 57 us; skew 500 ppm
Sep 28 22:49:19 dist-test-slave-56vw krb5kdc[22413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635759, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:19.457008 22426 init.cc:377] Logged in from keytab as kudu/127.16.121.190@KRBTEST.COM (short username kudu)
I20260928 22:49:19.457592 22426 webserver.cc:460] Webserver started at http://127.16.121.190:33015/ using document root <none> and password file <none>
I20260928 22:49:19.457870 22426 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:19.457947 22426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:19.458112 22426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:19.459332 22426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "000890ce720244edbe42e604be0664f2"
format_stamp: "Formatted at 2026-09-28 22:49:19 on dist-test-slave-56vw"
server_key: "944eda76a9422d58292c3ebf74d6d8e9"
server_key_iv: "f7151ac2aae97051154bfdc5b1280abb"
server_key_version: "encryptionkey@0"
I20260928 22:49:19.459735 22426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "000890ce720244edbe42e604be0664f2"
format_stamp: "Formatted at 2026-09-28 22:49:19 on dist-test-slave-56vw"
server_key: "944eda76a9422d58292c3ebf74d6d8e9"
server_key_iv: "f7151ac2aae97051154bfdc5b1280abb"
server_key_version: "encryptionkey@0"
I20260928 22:49:19.461223 22426 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:49:19.462289 22441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:19.462791 22426 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:19.462929 22426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "000890ce720244edbe42e604be0664f2"
format_stamp: "Formatted at 2026-09-28 22:49:19 on dist-test-slave-56vw"
server_key: "944eda76a9422d58292c3ebf74d6d8e9"
server_key_iv: "f7151ac2aae97051154bfdc5b1280abb"
server_key_version: "encryptionkey@0"
I20260928 22:49:19.463025 22426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:19.490777 22426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:19.493350 22426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:19.493487 22426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:19.498050 22426 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.190:37045
I20260928 22:49:19.498126 22493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.190:37045 every 8 connection(s)
I20260928 22:49:19.498521 22426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:49:19.499622 22494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:19.502192 22494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 000890ce720244edbe42e604be0664f2: Bootstrap starting.
I20260928 22:49:19.503163 22494 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 000890ce720244edbe42e604be0664f2: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:19.503507 22494 log.cc:824] T 00000000000000000000000000000000 P 000890ce720244edbe42e604be0664f2: Log is configured to *not* fsync() on all Append() calls
I20260928 22:49:19.504372 22494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 000890ce720244edbe42e604be0664f2: No bootstrap required, opened a new log
I20260928 22:49:19.505538 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 22426
I20260928 22:49:19.505623 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:49:19.505807 16870 external_mini_cluster.cc:1428] Setting key be64f05c83680772030614955efcf2c3
I20260928 22:49:19.506712 22494 raft_consensus.cc:348] T 00000000000000000000000000000000 P 000890ce720244edbe42e604be0664f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "000890ce720244edbe42e604be0664f2" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37045 } }
I20260928 22:49:19.506896 22494 raft_consensus.cc:374] T 00000000000000000000000000000000 P 000890ce720244edbe42e604be0664f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:19.506960 22494 raft_consensus.cc:729] T 00000000000000000000000000000000 P 000890ce720244edbe42e604be0664f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 000890ce720244edbe42e604be0664f2, State: Initialized, Role: FOLLOWER
I20260928 22:49:19.507130 22494 consensus_queue.cc:260] T 00000000000000000000000000000000 P 000890ce720244edbe42e604be0664f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "000890ce720244edbe42e604be0664f2" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37045 } }
I20260928 22:49:19.507233 22494 raft_consensus.cc:388] T 00000000000000000000000000000000 P 000890ce720244edbe42e604be0664f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:49:19.507280 22494 raft_consensus.cc:482] T 00000000000000000000000000000000 P 000890ce720244edbe42e604be0664f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:49:19.507337 22494 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 000890ce720244edbe42e604be0664f2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:19.508318 22494 raft_consensus.cc:504] T 00000000000000000000000000000000 P 000890ce720244edbe42e604be0664f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "000890ce720244edbe42e604be0664f2" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37045 } }
I20260928 22:49:19.508468 22494 leader_election.cc:302] T 00000000000000000000000000000000 P 000890ce720244edbe42e604be0664f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 000890ce720244edbe42e604be0664f2; no voters:
I20260928 22:49:19.508697 22494 leader_election.cc:288] T 00000000000000000000000000000000 P 000890ce720244edbe42e604be0664f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:49:19.508730 22499 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 000890ce720244edbe42e604be0664f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:49:19.508849 22499 raft_consensus.cc:686] T 00000000000000000000000000000000 P 000890ce720244edbe42e604be0664f2 [term 1 LEADER]: Becoming Leader. State: Replica: 000890ce720244edbe42e604be0664f2, State: Running, Role: LEADER
I20260928 22:49:19.509001 22494 sys_catalog.cc:564] T 00000000000000000000000000000000 P 000890ce720244edbe42e604be0664f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:49:19.508986 22499 consensus_queue.cc:237] T 00000000000000000000000000000000 P 000890ce720244edbe42e604be0664f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "000890ce720244edbe42e604be0664f2" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37045 } }
I20260928 22:49:19.509343 22499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 000890ce720244edbe42e604be0664f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 000890ce720244edbe42e604be0664f2. Latest consensus state: current_term: 1 leader_uuid: "000890ce720244edbe42e604be0664f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "000890ce720244edbe42e604be0664f2" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37045 } } }
I20260928 22:49:19.509363 22500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 000890ce720244edbe42e604be0664f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "000890ce720244edbe42e604be0664f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "000890ce720244edbe42e604be0664f2" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37045 } } }
I20260928 22:49:19.509550 22500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 000890ce720244edbe42e604be0664f2 [sys.catalog]: This master's current role is: LEADER
I20260928 22:49:19.509734 22499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 000890ce720244edbe42e604be0664f2 [sys.catalog]: This master's current role is: LEADER
I20260928 22:49:19.509903 22508 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:49:19.510471 22508 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:49:19.512220 22508 catalog_manager.cc:1347] Generated new cluster ID: c8d57955d352475d8373d4e92eebce93
I20260928 22:49:19.512285 22508 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 22:49:19 dist-test-slave-56vw krb5kdc[22413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:49:19.518056 22498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:19.506426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56690 (local address 127.16.121.190:37045)
0928 22:49:19.506622 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:19.506633 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:19.506672 (+ 39us) server_negotiation.cc:407] Connection header received
0928 22:49:19.506791 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:19.506797 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:19.506843 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:19.506990 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:19.507788 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:19.508942 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:19.510180 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:19.510964 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:19.511131 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:19.514067 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:19.514072 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:19.514072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:19.514086 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:19.516197 (+ 2111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:19.516623 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:19.516626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:19.516627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:19.516773 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:19.516994 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:19.516995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:19.516995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:19.517364 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:19.517578 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:19.517665 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 22:49:19.517829 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":97,"threads_started":1}
I20260928 22:49:19.541693 22508 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:49:19.542529 22508 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:49:19.546586 22508 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 000890ce720244edbe42e604be0664f2: Generated new TSK 0
I20260928 22:49:19.546882 22508 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.121.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab.
Entry for principal kudu/127.16.121.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab.
I20260928 22:49:19.604665 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.129:0
--local_ip_for_outbound_sockets=127.16.121.129
--webserver_interface=127.16.121.129
--webserver_port=0
--tserver_master_addrs=127.16.121.190:37045
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab
--principal=kudu/127.16.121.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:34587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:49:19.720018 22520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:19.720273 22520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:19.720376 22520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:19.721827 22520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:19.721885 22520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:19.721917 22520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.129
I20260928 22:49:19.723703 22520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:34587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab
--principal=kudu/127.16.121.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.121.129
--webserver_port=0
--tserver_master_addrs=127.16.121.190:37045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.129
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:19.723938 22520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:19.724138 22520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:19.729758 22528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:19.729743 22526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:19.729743 22525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:19.730613 22520 server_base.cc:1034] running on GCE node
I20260928 22:49:19.730883 22520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:19.731204 22520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:19.732384 22520 hybrid_clock.cc:648] HybridClock initialized: now 1790635759732376 us; error 38 us; skew 500 ppm
Sep 28 22:49:19 dist-test-slave-56vw krb5kdc[22413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635759, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:19.740305 22520 init.cc:377] Logged in from keytab as kudu/127.16.121.129@KRBTEST.COM (short username kudu)
I20260928 22:49:19.740989 22520 webserver.cc:460] Webserver started at http://127.16.121.129:33175/ using document root <none> and password file <none>
I20260928 22:49:19.741252 22520 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:19.741329 22520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:19.741501 22520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:19.742677 22520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-0/data/instance:
uuid: "5e8843a2872b4474b4940dc4e272da95"
format_stamp: "Formatted at 2026-09-28 22:49:19 on dist-test-slave-56vw"
server_key: "97bf4e08b72a614b1db36c5ff8de7f3f"
server_key_iv: "84c9e39ddfe34341ac88a541f6416d16"
server_key_version: "encryptionkey@0"
I20260928 22:49:19.743155 22520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance:
uuid: "5e8843a2872b4474b4940dc4e272da95"
format_stamp: "Formatted at 2026-09-28 22:49:19 on dist-test-slave-56vw"
server_key: "97bf4e08b72a614b1db36c5ff8de7f3f"
server_key_iv: "84c9e39ddfe34341ac88a541f6416d16"
server_key_version: "encryptionkey@0"
I20260928 22:49:19.744658 22520 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:19.745541 22535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:19.746047 22520 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:19.746188 22520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-0/wal
uuid: "5e8843a2872b4474b4940dc4e272da95"
format_stamp: "Formatted at 2026-09-28 22:49:19 on dist-test-slave-56vw"
server_key: "97bf4e08b72a614b1db36c5ff8de7f3f"
server_key_iv: "84c9e39ddfe34341ac88a541f6416d16"
server_key_version: "encryptionkey@0"
I20260928 22:49:19.746289 22520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:19.759943 22520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:19.762636 22520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:19.762840 22520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:19.763170 22520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:49:19.763540 22520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:49:19.763600 22520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:19.763644 22520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:49:19.763671 22520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:19.771375 22520 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.129:42247
I20260928 22:49:19.771481 22648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.129:42247 every 8 connection(s)
I20260928 22:49:19.771893 22520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
I20260928 22:49:19.773214 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 22520
I20260928 22:49:19.773344 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance
I20260928 22:49:19.773576 16870 external_mini_cluster.cc:1428] Setting key bd9564229d004b6137994675d2f45515
Sep 28 22:49:19 dist-test-slave-56vw krb5kdc[22413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635759, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.121.130@KRBTEST.COM; defaulting to no policy
I20260928 22:49:19.783936 22498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:19.773075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:54555 (local address 127.16.121.190:37045)
0928 22:49:19.773134 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:19.773136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:19.773365 (+ 229us) server_negotiation.cc:407] Connection header received
0928 22:49:19.774424 (+ 1059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:19.774426 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:19.774443 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:19.774779 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:19.775470 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:19.775979 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:19.777243 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:19.777596 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:19.777680 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:19.780410 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:19.780413 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:19.780413 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:19.780426 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:19.782197 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:19.782697 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:19.782698 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:19.782699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:19.782767 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:19.783145 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:19.783146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:19.783147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:19.783300 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:19.783374 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:19.783725 (+ 351us) server_negotiation.cc:299] Negotiation successful
0928 22:49:19.783833 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 22:49:19.784019 22651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:19.772696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37045 (local address 127.16.121.129:54555)
0928 22:49:19.773152 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:19.773200 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:19.774236 (+ 1036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:19.774649 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:19.774653 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:19.775003 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:19.775337 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:19.775344 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:19.776082 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:19.776083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:19.777135 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:19.777138 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:19.777697 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:19.777698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:19.777809 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:19.778418 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:19.778427 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:19.780221 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:19.782341 (+ 2120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:19.782349 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:19.782351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:19.782601 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:19.782878 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:19.782880 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:19.782880 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:19.783003 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:19.783378 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:19.783385 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:19.783531 (+ 146us) client_negotiation.cc:769] Sending connection context
0928 22:49:19.783657 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 22:49:19.783816 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":97,"threads_started":1}
I20260928 22:49:19.784530 22649 heartbeater.cc:347] Connected to a master server at 127.16.121.190:37045
I20260928 22:49:19.784634 22649 heartbeater.cc:464] Registering TS with master...
I20260928 22:49:19.784788 22649 heartbeater.cc:511] Master 127.16.121.190:37045 requested a full tablet report, sending...
I20260928 22:49:19.785394 22458 ts_manager.cc:194] Registered new tserver with Master: 5e8843a2872b4474b4940dc4e272da95 (127.16.121.129:42247)
I20260928 22:49:19.786163 22458 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.129@KRBTEST.COM'} at 127.16.121.129:54555
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab.
Entry for principal kudu/127.16.121.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab.
I20260928 22:49:19.804500 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.130:0
--local_ip_for_outbound_sockets=127.16.121.130
--webserver_interface=127.16.121.130
--webserver_port=0
--tserver_master_addrs=127.16.121.190:37045
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab
--principal=kudu/127.16.121.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:34587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:49:19.894081 22654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:19.894328 22654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:19.894356 22654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:19.895833 22654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:19.895900 22654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:19.895931 22654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.130
I20260928 22:49:19.897678 22654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:34587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab
--principal=kudu/127.16.121.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.121.130
--webserver_port=0
--tserver_master_addrs=127.16.121.190:37045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.130
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:19.897970 22654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:19.898253 22654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:19.904374 22659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:19.904436 22654 server_base.cc:1034] running on GCE node
W20260928 22:49:19.904487 22662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:19.904505 22660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:19.904950 22654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:19.905200 22654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:19.906360 22654 hybrid_clock.cc:648] HybridClock initialized: now 1790635759906346 us; error 36 us; skew 500 ppm
Sep 28 22:49:19 dist-test-slave-56vw krb5kdc[22413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635759, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:19.909564 22654 init.cc:377] Logged in from keytab as kudu/127.16.121.130@KRBTEST.COM (short username kudu)
I20260928 22:49:19.910091 22654 webserver.cc:460] Webserver started at http://127.16.121.130:39091/ using document root <none> and password file <none>
I20260928 22:49:19.910425 22654 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:19.910506 22654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:19.910610 22654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:19.912048 22654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-1/data/instance:
uuid: "838a2b7bde934cf3a00d90eb33757d0a"
format_stamp: "Formatted at 2026-09-28 22:49:19 on dist-test-slave-56vw"
server_key: "6cf4982c5637ee560bacad3bf920dd31"
server_key_iv: "56bcd17714236a0991d50ce78f5f7bcc"
server_key_version: "encryptionkey@0"
I20260928 22:49:19.912473 22654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance:
uuid: "838a2b7bde934cf3a00d90eb33757d0a"
format_stamp: "Formatted at 2026-09-28 22:49:19 on dist-test-slave-56vw"
server_key: "6cf4982c5637ee560bacad3bf920dd31"
server_key_iv: "56bcd17714236a0991d50ce78f5f7bcc"
server_key_version: "encryptionkey@0"
I20260928 22:49:19.914052 22654 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:49:19.915114 22669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:19.915321 22654 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:19.915419 22654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-1/wal
uuid: "838a2b7bde934cf3a00d90eb33757d0a"
format_stamp: "Formatted at 2026-09-28 22:49:19 on dist-test-slave-56vw"
server_key: "6cf4982c5637ee560bacad3bf920dd31"
server_key_iv: "56bcd17714236a0991d50ce78f5f7bcc"
server_key_version: "encryptionkey@0"
I20260928 22:49:19.915585 22654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:19.926882 22654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:19.929667 22654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:19.929857 22654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:19.930147 22654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:49:19.930491 22654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:49:19.930549 22654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:19.930594 22654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:49:19.930620 22654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:19.936723 22654 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.130:45565
I20260928 22:49:19.936770 22782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.130:45565 every 8 connection(s)
I20260928 22:49:19.937168 22654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
I20260928 22:49:19.940512 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 22654
I20260928 22:49:19.940627 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance
I20260928 22:49:19.940886 16870 external_mini_cluster.cc:1428] Setting key 46deb2067c1dc47c21868711d30af71b
Sep 28 22:49:19 dist-test-slave-56vw krb5kdc[22413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635759, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.130@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:49:19.950105 22498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:19.938264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:38309 (local address 127.16.121.190:37045)
0928 22:49:19.938404 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:19.938406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:19.938641 (+ 235us) server_negotiation.cc:407] Connection header received
0928 22:49:19.939819 (+ 1178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:19.939821 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:19.939835 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:19.939912 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:19.940757 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:19.941482 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:19.942720 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:19.943103 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:19.943187 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:19.946173 (+ 2986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:19.946175 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:19.946175 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:19.946185 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:19.948153 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:19.948899 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:19.948901 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:19.948902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:19.949021 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:19.949350 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:19.949351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:19.949352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:19.949496 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:19.949575 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:19.949903 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 22:49:19.950009 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260928 22:49:19.950341 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:19.938221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37045 (local address 127.16.121.130:38309)
0928 22:49:19.938489 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:19.938549 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:19.939677 (+ 1128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:19.940002 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:19.940006 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:19.940309 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:19.940649 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:19.940657 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:19.941610 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:19.941611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:19.942611 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:19.942614 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:19.943230 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:19.943232 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:19.943544 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:19.944240 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:19.944252 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:19.946051 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:19.948431 (+ 2380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:19.948484 (+ 53us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:19.948486 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:19.948761 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:19.949114 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:19.949115 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:19.949115 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:19.949257 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:19.949575 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:19.949584 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:19.949754 (+ 170us) client_negotiation.cc:769] Sending connection context
0928 22:49:19.949870 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 22:49:19.950057 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":103,"threads_started":1}
I20260928 22:49:19.950860 22783 heartbeater.cc:347] Connected to a master server at 127.16.121.190:37045
WARNING: no policy specified for kudu/127.16.121.131@KRBTEST.COM; defaulting to no policy
I20260928 22:49:19.950968 22783 heartbeater.cc:464] Registering TS with master...
I20260928 22:49:19.951220 22783 heartbeater.cc:511] Master 127.16.121.190:37045 requested a full tablet report, sending...
I20260928 22:49:19.951663 22458 ts_manager.cc:194] Registered new tserver with Master: 838a2b7bde934cf3a00d90eb33757d0a (127.16.121.130:45565)
I20260928 22:49:19.952073 22458 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.130@KRBTEST.COM'} at 127.16.121.130:38309
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab.
Entry for principal kudu/127.16.121.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab.
I20260928 22:49:19.971276 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.131:0
--local_ip_for_outbound_sockets=127.16.121.131
--webserver_interface=127.16.121.131
--webserver_port=0
--tserver_master_addrs=127.16.121.190:37045
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab
--principal=kudu/127.16.121.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:34587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:49:20.067224 22788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:20.067464 22788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:20.067494 22788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:20.068843 22788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:20.068886 22788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:20.068918 22788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.131
I20260928 22:49:20.070549 22788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:34587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab
--principal=kudu/127.16.121.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.121.131
--webserver_port=0
--tserver_master_addrs=127.16.121.190:37045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.131
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:20.070768 22788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:20.070981 22788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 22:49:20.076675 22788 server_base.cc:1034] running on GCE node
W20260928 22:49:20.076620 22796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:20.076682 22794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:20.076624 22793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:20.077114 22788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:20.077389 22788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:20.078584 22788 hybrid_clock.cc:648] HybridClock initialized: now 1790635760078565 us; error 33 us; skew 500 ppm
Sep 28 22:49:20 dist-test-slave-56vw krb5kdc[22413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635760, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:20.081717 22788 init.cc:377] Logged in from keytab as kudu/127.16.121.131@KRBTEST.COM (short username kudu)
I20260928 22:49:20.082239 22788 webserver.cc:460] Webserver started at http://127.16.121.131:44605/ using document root <none> and password file <none>
I20260928 22:49:20.082517 22788 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:20.082588 22788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:20.082741 22788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:20.083928 22788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-2/data/instance:
uuid: "d2978ce370b1407c825c064463a368f7"
format_stamp: "Formatted at 2026-09-28 22:49:20 on dist-test-slave-56vw"
server_key: "68a2d0b5b83e124246373e6dbf461017"
server_key_iv: "c238bcf72bcd93cbc216f5a374a0bcd3"
server_key_version: "encryptionkey@0"
I20260928 22:49:20.084292 22788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance:
uuid: "d2978ce370b1407c825c064463a368f7"
format_stamp: "Formatted at 2026-09-28 22:49:20 on dist-test-slave-56vw"
server_key: "68a2d0b5b83e124246373e6dbf461017"
server_key_iv: "c238bcf72bcd93cbc216f5a374a0bcd3"
server_key_version: "encryptionkey@0"
I20260928 22:49:20.085770 22788 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 22:49:20.086802 22803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:20.087092 22788 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:49:20.087203 22788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-2/wal
uuid: "d2978ce370b1407c825c064463a368f7"
format_stamp: "Formatted at 2026-09-28 22:49:20 on dist-test-slave-56vw"
server_key: "68a2d0b5b83e124246373e6dbf461017"
server_key_iv: "c238bcf72bcd93cbc216f5a374a0bcd3"
server_key_version: "encryptionkey@0"
I20260928 22:49:20.087287 22788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:20.106237 22788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:20.109512 22788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:20.109647 22788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:20.109921 22788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:49:20.110286 22788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:49:20.110322 22788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:20.110345 22788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:49:20.110358 22788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:20.116989 22788 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.131:41755
I20260928 22:49:20.117185 22916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.131:41755 every 8 connection(s)
I20260928 22:49:20.117497 22788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
Sep 28 22:49:20 dist-test-slave-56vw krb5kdc[22413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635760, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.131@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:49:20.127693 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 22788
I20260928 22:49:20.127836 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance
I20260928 22:49:20.128021 16870 external_mini_cluster.cc:1428] Setting key 4288fa9f921438686c1d1447956c3a3d
I20260928 22:49:20.129738 22919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:20.118499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37045 (local address 127.16.121.131:42973)
0928 22:49:20.118774 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:20.118826 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:20.120006 (+ 1180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:20.120333 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:20.120337 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:20.120670 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:20.120895 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:20.120899 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:20.121710 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:20.121711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:20.122674 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:20.122677 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:20.123200 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:20.123201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:20.123359 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:20.124015 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:20.124025 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:20.125832 (+ 1807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:20.127967 (+ 2135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:20.127976 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:20.127979 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:20.128329 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:20.128609 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:20.128611 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:20.128611 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:20.128781 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:20.129121 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:20.129129 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:20.129295 (+ 166us) client_negotiation.cc:769] Sending connection context
0928 22:49:20.129400 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 22:49:20.129549 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":149,"threads_started":1}
I20260928 22:49:20.129765 22498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:20.118642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:42973 (local address 127.16.121.190:37045)
0928 22:49:20.118734 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:20.118743 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:20.118922 (+ 179us) server_negotiation.cc:407] Connection header received
0928 22:49:20.120151 (+ 1229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:20.120153 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:20.120168 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:20.120234 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:20.121022 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:20.121586 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:20.122794 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:20.123029 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:20.123186 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:20.125987 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:20.125989 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:20.125990 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:20.126000 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:20.127811 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:20.128424 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:20.128425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:20.128426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:20.128509 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:20.128882 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:20.128883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:20.128883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:20.129044 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:20.129122 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:20.129606 (+ 484us) server_negotiation.cc:299] Negotiation successful
0928 22:49:20.129699 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 22:49:20.130378 22917 heartbeater.cc:347] Connected to a master server at 127.16.121.190:37045
I20260928 22:49:20.130522 22917 heartbeater.cc:464] Registering TS with master...
I20260928 22:49:20.130744 22917 heartbeater.cc:511] Master 127.16.121.190:37045 requested a full tablet report, sending...
I20260928 22:49:20.131289 22458 ts_manager.cc:194] Registered new tserver with Master: d2978ce370b1407c825c064463a368f7 (127.16.121.131:41755)
I20260928 22:49:20.131706 22458 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.131@KRBTEST.COM'} at 127.16.121.131:42973
I20260928 22:49:20.140628 16870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/krb5kdc/test-admin.keytab.
Sep 28 22:49:20 dist-test-slave-56vw krb5kdc[22413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:20.161202 16870 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 22:49:20.163386 22924 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:20.162358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59608 (local address 127.16.121.129:42247)
0928 22:49:20.162640 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:20.162645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:20.162656 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:49:20.162686 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:20.162689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:20.162709 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:20.162784 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:20.163274 (+ 490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59608: BlockingRecv error: recv got EOF from 127.0.0.1:59608 (error 108)
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":189,"threads_started":1}
W20260928 22:49:20.163416 16870 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/krb5kdc/krb5cc' of type FILE
Sep 28 22:49:20 dist-test-slave-56vw krb5kdc[22413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:20.164646 16870 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 22:49:20 dist-test-slave-56vw krb5kdc[22413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:20.165724 16870 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 22:49:20 dist-test-slave-56vw krb5kdc[22413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.121.129@KRBTEST.COM
I20260928 22:49:20.175422 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:20.166647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59616 (local address 127.16.121.129:42247)
0928 22:49:20.166728 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:20.166730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:20.166740 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:49:20.166766 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:20.166767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:20.166778 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:20.166910 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:20.167499 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:20.168458 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:20.169314 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:20.169700 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:20.169768 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:20.172274 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:20.172285 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:20.172286 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:20.172295 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:20.174116 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:20.174544 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:20.174547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:20.174548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:20.174603 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:20.174779 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:20.174780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:20.174820 (+ 40us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:20.174964 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:20.175136 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:20.175248 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 22:49:20.175326 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 22:49:20.177858 22924 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:20.177097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59628 (local address 127.16.121.129:42247)
0928 22:49:20.177162 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:20.177164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:20.177230 (+ 66us) server_negotiation.cc:407] Connection header received
0928 22:49:20.177294 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:20.177295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:20.177306 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:20.177352 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:20.177808 (+ 456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59628: BlockingRecv error: recv got EOF from 127.0.0.1:59628 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260928 22:49:20.178107 16870 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/krb5kdc/krb5cc)
Sep 28 22:49:20 dist-test-slave-56vw krb5kdc[22413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:20.179360 16870 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 22:49:20 dist-test-slave-56vw krb5kdc[22413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:20.180346 16870 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 22:49:20 dist-test-slave-56vw krb5kdc[22413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.121.129@KRBTEST.COM
I20260928 22:49:20.188745 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:20.180992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59638 (local address 127.16.121.129:42247)
0928 22:49:20.181036 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:20.181038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:20.181075 (+ 37us) server_negotiation.cc:407] Connection header received
0928 22:49:20.181144 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:20.181145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:20.181154 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:20.181210 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:20.181771 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:20.182388 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:20.183370 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:20.183585 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:20.183647 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:20.185966 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:20.185967 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:20.185967 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:20.185975 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:20.187673 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:20.188045 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:20.188046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:20.188046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:20.188112 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:20.188264 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:20.188265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:20.188265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:20.188437 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:20.188520 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:20.188603 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 22:49:20.188674 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 22:49:20.190762 22924 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:20.189976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59654 (local address 127.16.121.129:42247)
0928 22:49:20.190065 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:20.190066 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:20.190174 (+ 108us) server_negotiation.cc:407] Connection header received
0928 22:49:20.190244 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:20.190245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:20.190256 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:20.190317 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:20.190698 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59654: BlockingRecv error: recv got EOF from 127.0.0.1:59654 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260928 22:49:20.190819 16870 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790635733329745-16870-0/krb5kdc/krb5cc)
Sep 28 22:49:20 dist-test-slave-56vw krb5kdc[22413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:20.192024 16870 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 22:49:20 dist-test-slave-56vw krb5kdc[22413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:20.193070 16870 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 22:49:20 dist-test-slave-56vw krb5kdc[22413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.121.129@KRBTEST.COM
I20260928 22:49:20.201313 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:20.193638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59656 (local address 127.16.121.129:42247)
0928 22:49:20.193706 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:20.193707 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:20.193793 (+ 86us) server_negotiation.cc:407] Connection header received
0928 22:49:20.193818 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:20.193819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:20.193862 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:20.193919 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:20.194314 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:20.194875 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:20.195666 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:20.195896 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:20.195957 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:20.198454 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:20.198456 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:20.198456 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:20.198464 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:20.200042 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:20.200498 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:20.200499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:20.200500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:20.200555 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:20.200728 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:20.200728 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:20.200729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:20.200833 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:20.200906 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:20.201081 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 22:49:20.201148 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 22:49:20.202144 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 22520
I20260928 22:49:20.206923 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 22654
I20260928 22:49:20.212393 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 22788
I20260928 22:49:20.217552 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 22426
2026-09-28T22:49:20Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2094 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:49:20 dist-test-slave-56vw krb5kdc[22937](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:49:20 dist-test-slave-56vw krb5kdc[22937](info): set up 2 sockets
Sep 28 22:49:20 dist-test-slave-56vw krb5kdc[22937](info): commencing operation
krb5kdc: starting...
W20260928 22:49:21.283640 16870 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Sep 28 22:49:21 dist-test-slave-56vw krb5kdc[22937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:49:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:49:21Z Disabled control of system clock
WARNING: no policy specified for oryx/127.16.121.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.121.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.121.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.190.keytab.
Entry for principal oryx/127.16.121.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.190.keytab.
I20260928 22:49:21.416483 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:39331
--webserver_interface=127.16.121.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.190.keytab
--principal=oryx/127.16.121.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:33939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:39331
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:49:21.499933 22950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:21.500164 22950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:21.500190 22950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:21.501418 22950 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:49:21.501461 22950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:21.501477 22950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:21.501488 22950 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:49:21.501500 22950 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:49:21.503013 22950 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:33939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:39331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.190.keytab
--principal=oryx/127.16.121.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:39331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.121.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:21.503294 22950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:21.503475 22950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:21.508569 22956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:21.508718 22955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:21.508847 22958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:21.509219 22950 server_base.cc:1034] running on GCE node
I20260928 22:49:21.509579 22950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:21.509930 22950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:21.511126 22950 hybrid_clock.cc:648] HybridClock initialized: now 1790635761511086 us; error 52 us; skew 500 ppm
Sep 28 22:49:21 dist-test-slave-56vw krb5kdc[22937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635761, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.121.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:21.514065 22950 init.cc:377] Logged in from keytab as oryx/127.16.121.190@KRBTEST.COM (short username oryx)
I20260928 22:49:21.514647 22950 webserver.cc:460] Webserver started at http://127.16.121.190:37975/ using document root <none> and password file <none>
I20260928 22:49:21.514914 22950 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:21.514968 22950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:21.515138 22950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:21.516291 22950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "6c2ca4515ad9482a8d102db695d461ec"
format_stamp: "Formatted at 2026-09-28 22:49:21 on dist-test-slave-56vw"
server_key: "9c0ab7dff327117eb6b60d1ba09a7165"
server_key_iv: "3e9763dade6759348137942e1839c658"
server_key_version: "encryptionkey@0"
I20260928 22:49:21.516690 22950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "6c2ca4515ad9482a8d102db695d461ec"
format_stamp: "Formatted at 2026-09-28 22:49:21 on dist-test-slave-56vw"
server_key: "9c0ab7dff327117eb6b60d1ba09a7165"
server_key_iv: "3e9763dade6759348137942e1839c658"
server_key_version: "encryptionkey@0"
I20260928 22:49:21.518136 22950 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:49:21.519119 22965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:21.519295 22950 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:21.519465 22950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "6c2ca4515ad9482a8d102db695d461ec"
format_stamp: "Formatted at 2026-09-28 22:49:21 on dist-test-slave-56vw"
server_key: "9c0ab7dff327117eb6b60d1ba09a7165"
server_key_iv: "3e9763dade6759348137942e1839c658"
server_key_version: "encryptionkey@0"
I20260928 22:49:21.519554 22950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:21.536504 22950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:21.539340 22950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:21.539533 22950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:21.544055 22950 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.190:39331
I20260928 22:49:21.544098 23017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.190:39331 every 8 connection(s)
I20260928 22:49:21.544510 22950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:49:21.545667 23018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:21.548218 23018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec: Bootstrap starting.
I20260928 22:49:21.548955 23018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:21.549275 23018 log.cc:824] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec: Log is configured to *not* fsync() on all Append() calls
I20260928 22:49:21.550125 23018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec: No bootstrap required, opened a new log
I20260928 22:49:21.552043 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 22950
I20260928 22:49:21.552160 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:49:21.552181 23018 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } }
I20260928 22:49:21.552351 16870 external_mini_cluster.cc:1428] Setting key b6209df5d90d3b549c9c27318ab05b4f
I20260928 22:49:21.552385 23018 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:21.552425 23018 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c2ca4515ad9482a8d102db695d461ec, State: Initialized, Role: FOLLOWER
I20260928 22:49:21.552580 23018 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } }
I20260928 22:49:21.552676 23018 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:49:21.552712 23018 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:49:21.552799 23018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:21.553645 23018 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } }
I20260928 22:49:21.553757 23018 leader_election.cc:302] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c2ca4515ad9482a8d102db695d461ec; no voters:
I20260928 22:49:21.553930 23018 leader_election.cc:288] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:49:21.554009 23023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:49:21.554212 23023 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec [term 1 LEADER]: Becoming Leader. State: Replica: 6c2ca4515ad9482a8d102db695d461ec, State: Running, Role: LEADER
I20260928 22:49:21.554246 23018 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:49:21.554342 23023 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } }
I20260928 22:49:21.554708 23024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c2ca4515ad9482a8d102db695d461ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } }
I20260928 22:49:21.554724 23025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c2ca4515ad9482a8d102db695d461ec. Latest consensus state: current_term: 1 leader_uuid: "6c2ca4515ad9482a8d102db695d461ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } }
I20260928 22:49:21.554812 23024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec [sys.catalog]: This master's current role is: LEADER
I20260928 22:49:21.554817 23025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec [sys.catalog]: This master's current role is: LEADER
I20260928 22:49:21.555076 23029 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:49:21.555692 23029 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:49:21.557442 23029 catalog_manager.cc:1347] Generated new cluster ID: f581d50fabd146eab37464bd687b1b1c
I20260928 22:49:21.557544 23029 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 22:49:21 dist-test-slave-56vw krb5kdc[22937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM
I20260928 22:49:21.564116 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:21.552842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46882 (local address 127.16.121.190:39331)
0928 22:49:21.553135 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:21.553145 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:21.553187 (+ 42us) server_negotiation.cc:407] Connection header received
0928 22:49:21.553634 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:21.553645 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:21.553719 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:21.553891 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:21.554879 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:21.556548 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:21.557548 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:21.558028 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:21.558162 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:21.560648 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:21.560653 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:21.560653 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:21.560667 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:21.562420 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:21.562884 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:21.562888 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:21.562888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:21.563009 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:21.563263 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:21.563264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:21.563264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:21.563536 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:21.563716 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:21.563796 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 22:49:21.563943 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":175,"threads_started":1}
I20260928 22:49:21.580184 23029 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:49:21.580770 23029 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:49:21.584405 23029 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec: Generated new TSK 0
I20260928 22:49:21.584649 23029 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.16.121.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.121.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.121.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.129.keytab.
Entry for principal oryx/127.16.121.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.129.keytab.
I20260928 22:49:21.626399 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.129:0
--local_ip_for_outbound_sockets=127.16.121.129
--webserver_interface=127.16.121.129
--webserver_port=0
--tserver_master_addrs=127.16.121.190:39331
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.129.keytab
--principal=oryx/127.16.121.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:33939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:49:21.724658 23044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:21.724891 23044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:21.724918 23044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:21.726086 23044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:21.726130 23044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:21.726159 23044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.129
I20260928 22:49:21.727648 23044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:33939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.129.keytab
--principal=oryx/127.16.121.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.121.129
--webserver_port=0
--tserver_master_addrs=127.16.121.190:39331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.129
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:21.727849 23044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:21.728039 23044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:21.733458 23050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:21.733458 23049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:21.733485 23052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:21.733913 23044 server_base.cc:1034] running on GCE node
I20260928 22:49:21.734176 23044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:21.734473 23044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:21.735647 23044 hybrid_clock.cc:648] HybridClock initialized: now 1790635761735635 us; error 33 us; skew 500 ppm
Sep 28 22:49:21 dist-test-slave-56vw krb5kdc[22937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635761, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.121.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:21.738271 23044 init.cc:377] Logged in from keytab as oryx/127.16.121.129@KRBTEST.COM (short username oryx)
I20260928 22:49:21.738873 23044 webserver.cc:460] Webserver started at http://127.16.121.129:44831/ using document root <none> and password file <none>
I20260928 22:49:21.739183 23044 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:21.739259 23044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:21.739420 23044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:21.740592 23044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-0/data/instance:
uuid: "fadc366fe3d44e068aee86188773c207"
format_stamp: "Formatted at 2026-09-28 22:49:21 on dist-test-slave-56vw"
server_key: "b04b8c9711778ac49b33251cbd0659d8"
server_key_iv: "5a6cf68701a29655d6dcb561f5a62738"
server_key_version: "encryptionkey@0"
I20260928 22:49:21.740976 23044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance:
uuid: "fadc366fe3d44e068aee86188773c207"
format_stamp: "Formatted at 2026-09-28 22:49:21 on dist-test-slave-56vw"
server_key: "b04b8c9711778ac49b33251cbd0659d8"
server_key_iv: "5a6cf68701a29655d6dcb561f5a62738"
server_key_version: "encryptionkey@0"
I20260928 22:49:21.742581 23044 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:49:21.743548 23059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:21.743801 23044 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 22:49:21.743928 23044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-0/wal
uuid: "fadc366fe3d44e068aee86188773c207"
format_stamp: "Formatted at 2026-09-28 22:49:21 on dist-test-slave-56vw"
server_key: "b04b8c9711778ac49b33251cbd0659d8"
server_key_iv: "5a6cf68701a29655d6dcb561f5a62738"
server_key_version: "encryptionkey@0"
I20260928 22:49:21.744011 23044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:21.770197 23044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:21.772766 23044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:21.772930 23044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:21.773193 23044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:49:21.773523 23044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:49:21.773558 23044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:21.773599 23044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:49:21.773624 23044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:21.779438 23044 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.129:42243
I20260928 22:49:21.779538 23172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.129:42243 every 8 connection(s)
I20260928 22:49:21.779872 23044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
I20260928 22:49:21.782608 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 23044
I20260928 22:49:21.782721 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance
I20260928 22:49:21.782920 16870 external_mini_cluster.cc:1428] Setting key 9a61a6bd3b5da0eeb1190f36972c73f2
Sep 28 22:49:21 dist-test-slave-56vw krb5kdc[22937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635761, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.121.129@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM
I20260928 22:49:21.791007 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:21.780630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:32863 (local address 127.16.121.190:39331)
0928 22:49:21.780714 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:21.780716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:21.781025 (+ 309us) server_negotiation.cc:407] Connection header received
0928 22:49:21.781524 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:21.781525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:21.781540 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:21.781599 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:21.782633 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:21.783405 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:21.784591 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:21.785001 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:21.785080 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:21.787725 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:21.787727 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:21.787727 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:21.787739 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:21.789325 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:21.789890 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:21.789892 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:21.789892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:21.789952 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:21.790278 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:21.790279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:21.790279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:21.790426 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:21.790504 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:21.790853 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 22:49:21.790926 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 22:49:21.791359 23175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:21.780589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:39331 (local address 127.16.121.129:32863)
0928 22:49:21.780854 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:21.780895 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:21.781422 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:21.781655 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:21.781658 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:21.782283 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:21.782505 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:21.782510 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:21.783501 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:21.783502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:21.784488 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:21.784490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:21.785125 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:21.785127 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:21.785378 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:21.785956 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:21.785967 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:21.787625 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:21.789421 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:21.789428 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:21.789431 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:21.789765 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:21.790032 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:21.790033 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:21.790034 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:21.790183 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:21.790508 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:21.790514 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:21.790695 (+ 181us) client_negotiation.cc:769] Sending connection context
0928 22:49:21.790828 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 22:49:21.791025 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":123,"threads_started":1}
WARNING: no policy specified for oryx/127.16.121.130@KRBTEST.COM; defaulting to no policy
I20260928 22:49:21.791954 23173 heartbeater.cc:347] Connected to a master server at 127.16.121.190:39331
I20260928 22:49:21.792076 23173 heartbeater.cc:464] Registering TS with master...
I20260928 22:49:21.792260 23173 heartbeater.cc:511] Master 127.16.121.190:39331 requested a full tablet report, sending...
I20260928 22:49:21.792830 22982 ts_manager.cc:194] Registered new tserver with Master: fadc366fe3d44e068aee86188773c207 (127.16.121.129:42243)
I20260928 22:49:21.793805 22982 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.121.129@KRBTEST.COM'} at 127.16.121.129:32863
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.121.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.121.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.130.keytab.
Entry for principal oryx/127.16.121.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.130.keytab.
I20260928 22:49:21.811739 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.130:0
--local_ip_for_outbound_sockets=127.16.121.130
--webserver_interface=127.16.121.130
--webserver_port=0
--tserver_master_addrs=127.16.121.190:39331
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.130.keytab
--principal=oryx/127.16.121.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:33939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:49:21.908052 23178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:21.908300 23178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:21.908340 23178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:21.909693 23178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:21.909741 23178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:21.909773 23178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.130
I20260928 22:49:21.911473 23178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:33939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.130.keytab
--principal=oryx/127.16.121.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.121.130
--webserver_port=0
--tserver_master_addrs=127.16.121.190:39331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.130
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:21.911696 23178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:21.911918 23178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:21.917606 23183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:21.917606 23184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:21.917624 23186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:21.917826 23178 server_base.cc:1034] running on GCE node
I20260928 22:49:21.918100 23178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:21.918366 23178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:21.919564 23178 hybrid_clock.cc:648] HybridClock initialized: now 1790635761919540 us; error 37 us; skew 500 ppm
Sep 28 22:49:21 dist-test-slave-56vw krb5kdc[22937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635761, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.121.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:21.922448 23178 init.cc:377] Logged in from keytab as oryx/127.16.121.130@KRBTEST.COM (short username oryx)
I20260928 22:49:21.922933 23178 webserver.cc:460] Webserver started at http://127.16.121.130:41013/ using document root <none> and password file <none>
I20260928 22:49:21.923184 23178 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:21.923231 23178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:21.923334 23178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:21.924396 23178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-1/data/instance:
uuid: "c12e3d60fcc24875801961d9ddb3fe63"
format_stamp: "Formatted at 2026-09-28 22:49:21 on dist-test-slave-56vw"
server_key: "3bd5ca716ca34ecc9bcb2e9a28d8b0b1"
server_key_iv: "42d8106f1b34d39dfd47e3cd86ad8760"
server_key_version: "encryptionkey@0"
I20260928 22:49:21.924754 23178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance:
uuid: "c12e3d60fcc24875801961d9ddb3fe63"
format_stamp: "Formatted at 2026-09-28 22:49:21 on dist-test-slave-56vw"
server_key: "3bd5ca716ca34ecc9bcb2e9a28d8b0b1"
server_key_iv: "42d8106f1b34d39dfd47e3cd86ad8760"
server_key_version: "encryptionkey@0"
I20260928 22:49:21.926182 23178 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:49:21.927172 23193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:21.927350 23178 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:21.927443 23178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-1/wal
uuid: "c12e3d60fcc24875801961d9ddb3fe63"
format_stamp: "Formatted at 2026-09-28 22:49:21 on dist-test-slave-56vw"
server_key: "3bd5ca716ca34ecc9bcb2e9a28d8b0b1"
server_key_iv: "42d8106f1b34d39dfd47e3cd86ad8760"
server_key_version: "encryptionkey@0"
I20260928 22:49:21.927526 23178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:21.938318 23178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:21.940740 23178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:21.940868 23178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:21.941078 23178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:49:21.941423 23178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:49:21.941458 23178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:21.941507 23178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:49:21.941533 23178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:21.947772 23178 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.130:43775
I20260928 22:49:21.947942 23306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.130:43775 every 8 connection(s)
I20260928 22:49:21.948189 23178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
Sep 28 22:49:21 dist-test-slave-56vw krb5kdc[22937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635761, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.121.130@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM
I20260928 22:49:21.957736 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 23178
I20260928 22:49:21.957875 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance
I20260928 22:49:21.958092 16870 external_mini_cluster.cc:1428] Setting key 11ffe05b468964e6b1e104b002f29a9b
I20260928 22:49:21.958884 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:21.949014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:44349 (local address 127.16.121.190:39331)
0928 22:49:21.949084 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:21.949087 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:21.949310 (+ 223us) server_negotiation.cc:407] Connection header received
0928 22:49:21.949836 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:21.949837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:21.949846 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:21.949901 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:21.950869 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:21.951424 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:21.952502 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:21.952696 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:21.952757 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:21.955606 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:21.955608 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:21.955608 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:21.955616 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:21.957131 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:21.957566 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:21.957568 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:21.957568 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:21.957635 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:21.957979 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:21.957981 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:21.957981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:21.958149 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:21.958224 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:21.958667 (+ 443us) server_negotiation.cc:299] Negotiation successful
0928 22:49:21.958776 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 22:49:21.959182 23309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:21.949007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:39331 (local address 127.16.121.130:44349)
0928 22:49:21.949181 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:21.949217 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:21.949716 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:21.949960 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:21.949964 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:21.950545 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:21.950741 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:21.950746 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:21.951516 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:21.951518 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:21.952393 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:21.952396 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:21.952798 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:21.952799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:21.952913 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:21.953667 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:21.953678 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:21.955509 (+ 1831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:21.957228 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:21.957234 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:21.957236 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:21.957484 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:21.957711 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:21.957712 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:21.957712 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:21.957822 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:21.958258 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:21.958266 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:21.958508 (+ 242us) client_negotiation.cc:769] Sending connection context
0928 22:49:21.958643 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 22:49:21.958854 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":91,"threads_started":1}
I20260928 22:49:21.959998 23307 heartbeater.cc:347] Connected to a master server at 127.16.121.190:39331
I20260928 22:49:21.960120 23307 heartbeater.cc:464] Registering TS with master...
I20260928 22:49:21.960314 23307 heartbeater.cc:511] Master 127.16.121.190:39331 requested a full tablet report, sending...
I20260928 22:49:21.960770 22982 ts_manager.cc:194] Registered new tserver with Master: c12e3d60fcc24875801961d9ddb3fe63 (127.16.121.130:43775)
I20260928 22:49:21.961158 22982 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.121.130@KRBTEST.COM'} at 127.16.121.130:44349
WARNING: no policy specified for oryx/127.16.121.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.121.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.121.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.131.keytab.
Entry for principal oryx/127.16.121.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.131.keytab.
I20260928 22:49:21.987555 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.131:0
--local_ip_for_outbound_sockets=127.16.121.131
--webserver_interface=127.16.121.131
--webserver_port=0
--tserver_master_addrs=127.16.121.190:39331
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.131.keytab
--principal=oryx/127.16.121.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:33939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:49:22.073884 23312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:22.074110 23312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:22.074136 23312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:22.075503 23312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:22.075556 23312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:22.075584 23312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.131
I20260928 22:49:22.077157 23312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:33939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.131.keytab
--principal=oryx/127.16.121.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.121.131
--webserver_port=0
--tserver_master_addrs=127.16.121.190:39331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.131
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:22.077440 23312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:22.077699 23312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 22:49:22.083173 23312 server_base.cc:1034] running on GCE node
W20260928 22:49:22.083143 23318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:22.083139 23320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:22.083143 23317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:22.083618 23312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:22.083887 23312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:22.085119 23312 hybrid_clock.cc:648] HybridClock initialized: now 1790635762085102 us; error 34 us; skew 500 ppm
Sep 28 22:49:22 dist-test-slave-56vw krb5kdc[22937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635762, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.121.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:22.087841 23312 init.cc:377] Logged in from keytab as oryx/127.16.121.131@KRBTEST.COM (short username oryx)
I20260928 22:49:22.088310 23312 webserver.cc:460] Webserver started at http://127.16.121.131:45939/ using document root <none> and password file <none>
I20260928 22:49:22.088500 23312 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:22.088541 23312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:22.088627 23312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:22.089699 23312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-2/data/instance:
uuid: "e1ab1de193eb40a9bb803104ee08c680"
format_stamp: "Formatted at 2026-09-28 22:49:22 on dist-test-slave-56vw"
server_key: "649056e8767f8f0a232830196af6a4fb"
server_key_iv: "a0fcf1ad1605c858ea0717032d9711e9"
server_key_version: "encryptionkey@0"
I20260928 22:49:22.090011 23312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance:
uuid: "e1ab1de193eb40a9bb803104ee08c680"
format_stamp: "Formatted at 2026-09-28 22:49:22 on dist-test-slave-56vw"
server_key: "649056e8767f8f0a232830196af6a4fb"
server_key_iv: "a0fcf1ad1605c858ea0717032d9711e9"
server_key_version: "encryptionkey@0"
I20260928 22:49:22.091419 23312 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:49:22.092238 23327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:22.092517 23312 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:49:22.092617 23312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-2/wal
uuid: "e1ab1de193eb40a9bb803104ee08c680"
format_stamp: "Formatted at 2026-09-28 22:49:22 on dist-test-slave-56vw"
server_key: "649056e8767f8f0a232830196af6a4fb"
server_key_iv: "a0fcf1ad1605c858ea0717032d9711e9"
server_key_version: "encryptionkey@0"
I20260928 22:49:22.092679 23312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:22.107275 23312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:22.109694 23312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:22.109838 23312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:22.110044 23312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:49:22.110405 23312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:49:22.110445 23312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:22.110468 23312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:49:22.110482 23312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:22.116866 23312 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.131:36269
I20260928 22:49:22.116948 23440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.131:36269 every 8 connection(s)
I20260928 22:49:22.117331 23312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
I20260928 22:49:22.123154 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 23312
I20260928 22:49:22.123234 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance
I20260928 22:49:22.123468 16870 external_mini_cluster.cc:1428] Setting key 4eba7cc25c55a52009021a3340dc8ed1
Sep 28 22:49:22 dist-test-slave-56vw krb5kdc[22937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635762, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.121.131@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM
I20260928 22:49:22.127676 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:22.118169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:55027 (local address 127.16.121.190:39331)
0928 22:49:22.118248 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:22.118250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:22.118469 (+ 219us) server_negotiation.cc:407] Connection header received
0928 22:49:22.119131 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:22.119133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:22.119149 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:22.119231 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:22.120304 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.120820 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.121947 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.122171 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.122247 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:22.124963 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:22.124964 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:22.124964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:22.124972 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:22.126288 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:22.126701 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:22.126702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:22.126703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:22.126758 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:22.126988 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:22.126989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:22.126989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:22.127177 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:22.127232 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:22.127526 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 22:49:22.127610 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 22:49:22.127808 23443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:22.118136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:39331 (local address 127.16.121.131:55027)
0928 22:49:22.118329 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:22.118368 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:22.118936 (+ 568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:22.119312 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:22.119316 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:22.119973 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:22.120181 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:22.120186 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.120940 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.120941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:22.121818 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:22.121821 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.122278 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.122279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:22.122387 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:22.122897 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:22.122905 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:22.124839 (+ 1934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:22.126367 (+ 1528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:22.126374 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:22.126376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:22.126627 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:22.126823 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:22.126823 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:22.126823 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:22.126924 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:22.127237 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:22.127242 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:22.127367 (+ 125us) client_negotiation.cc:769] Sending connection context
0928 22:49:22.127461 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 22:49:22.127591 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":112,"threads_started":1}
I20260928 22:49:22.128280 23441 heartbeater.cc:347] Connected to a master server at 127.16.121.190:39331
I20260928 22:49:22.128389 23441 heartbeater.cc:464] Registering TS with master...
I20260928 22:49:22.128571 23441 heartbeater.cc:511] Master 127.16.121.190:39331 requested a full tablet report, sending...
I20260928 22:49:22.129004 22982 ts_manager.cc:194] Registered new tserver with Master: e1ab1de193eb40a9bb803104ee08c680 (127.16.121.131:36269)
I20260928 22:49:22.129356 22982 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.121.131@KRBTEST.COM'} at 127.16.121.131:55027
I20260928 22:49:22.135327 16870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.16.121.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.121.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.121.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.189.keytab.
Entry for principal oryx/127.16.121.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.189.keytab.
I20260928 22:49:22.163508 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.189:46125
--webserver_interface=127.16.121.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.189.keytab
--principal=oryx/127.16.121.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:33939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:39331,127.16.121.189:46125
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:49:22.247859 23446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:22.248123 23446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:22.248164 23446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:22.249516 23446 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:49:22.249593 23446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:22.249619 23446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:22.249641 23446 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:49:22.249671 23446 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:49:22.251309 23446 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:33939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:39331,127.16.121.189:46125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.189.keytab
--principal=oryx/127.16.121.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.189:46125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.121.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
W20260928 22:49:22.251461 23446 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.16.121.190:39331,127.16.121.189:46125'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 22:49:22.251631 23446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:22.251873 23446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:22.257462 23452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:22.257508 23451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:22.257503 23454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:22.257553 23446 server_base.cc:1034] running on GCE node
I20260928 22:49:22.257938 23446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:22.258265 23446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:22.259398 23446 hybrid_clock.cc:648] HybridClock initialized: now 1790635762259373 us; error 43 us; skew 500 ppm
Sep 28 22:49:22 dist-test-slave-56vw krb5kdc[22937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635762, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.121.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:22.262089 23446 init.cc:377] Logged in from keytab as oryx/127.16.121.189@KRBTEST.COM (short username oryx)
I20260928 22:49:22.262667 23446 webserver.cc:460] Webserver started at http://127.16.121.189:41047/ using document root <none> and password file <none>
I20260928 22:49:22.262974 23446 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:22.263047 23446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:22.263227 23446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:22.264343 23446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-1/data/instance:
uuid: "a5fee57a42a14e1ab90bc3db2e453559"
format_stamp: "Formatted at 2026-09-28 22:49:22 on dist-test-slave-56vw"
server_key: "68a8843f90e7ea3ef4a8391b02023070"
server_key_iv: "48c920f8fa6f70879dc5c926998dd7ba"
server_key_version: "encryptionkey@0"
I20260928 22:49:22.264716 23446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-1/wal/instance:
uuid: "a5fee57a42a14e1ab90bc3db2e453559"
format_stamp: "Formatted at 2026-09-28 22:49:22 on dist-test-slave-56vw"
server_key: "68a8843f90e7ea3ef4a8391b02023070"
server_key_iv: "48c920f8fa6f70879dc5c926998dd7ba"
server_key_version: "encryptionkey@0"
I20260928 22:49:22.266109 23446 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:49:22.266885 23461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:22.267122 23446 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:49:22.267221 23446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-1/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-1/wal
uuid: "a5fee57a42a14e1ab90bc3db2e453559"
format_stamp: "Formatted at 2026-09-28 22:49:22 on dist-test-slave-56vw"
server_key: "68a8843f90e7ea3ef4a8391b02023070"
server_key_iv: "48c920f8fa6f70879dc5c926998dd7ba"
server_key_version: "encryptionkey@0"
I20260928 22:49:22.267299 23446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:22.280364 23446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:22.283274 23446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:22.283473 23446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:22.287761 23446 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.189:46125
I20260928 22:49:22.287822 23513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.189:46125 every 8 connection(s)
I20260928 22:49:22.288236 23446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-1/data/info.pb
I20260928 22:49:22.288827 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 23446
I20260928 22:49:22.289363 23514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:22.291659 23514 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 22:49:22 dist-test-slave-56vw krb5kdc[22937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.121.189@KRBTEST.COM
Sep 28 22:49:22 dist-test-slave-56vw krb5kdc[22937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635762, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.121.189@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM
I20260928 22:49:22.302078 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:22.292403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46888 (local address 127.16.121.190:39331)
0928 22:49:22.292498 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:22.292500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:22.292667 (+ 167us) server_negotiation.cc:407] Connection header received
0928 22:49:22.292745 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:22.292745 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:22.292756 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:22.292802 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:22.293437 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.294305 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.295484 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.295768 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.295830 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:22.298532 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:22.298534 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:22.298534 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:22.298551 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:22.300552 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:22.301069 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:22.301071 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:22.301071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:22.301153 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:22.301399 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:22.301400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:22.301400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:22.301601 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:22.301668 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:22.301893 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 22:49:22.301985 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 22:49:22.302289 23517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:22.289508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49822 (local address 127.16.121.189:46125)
0928 22:49:22.289707 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:22.289717 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:22.289752 (+ 35us) server_negotiation.cc:407] Connection header received
0928 22:49:22.290205 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:22.290212 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:22.290260 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:22.290424 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:22.291203 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.292269 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.293618 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.294242 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.294371 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:22.297770 (+ 3399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:22.297778 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:22.297779 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:22.297786 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:22.300268 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:22.300764 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:22.300767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:22.300767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:22.300851 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:22.301083 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:22.301083 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:22.301084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:22.301514 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:22.301685 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:22.301815 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 22:49:22.302019 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":147,"thread_start_us":89,"threads_started":1}
I20260928 22:49:22.302376 23518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:22.292340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:39331 (local address 127.0.0.1:46888)
0928 22:49:22.292532 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:22.292578 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:22.292677 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:22.292845 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:22.292849 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:22.293235 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:22.293322 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:22.293324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.294402 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.294403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:22.295381 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:22.295385 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.295858 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.295859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:22.295997 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:22.296598 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:22.296608 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:22.298433 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:22.300641 (+ 2208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:22.300649 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:22.300651 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:22.300973 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:22.301256 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:22.301257 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:22.301257 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:22.301318 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:22.301674 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:22.301680 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:22.301752 (+ 72us) client_negotiation.cc:769] Sending connection context
0928 22:49:22.301851 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 22:49:22.302212 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":164,"thread_start_us":109,"threads_started":1}
I20260928 22:49:22.303022 23514 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.121.189" port: 46125 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 22:49:22 dist-test-slave-56vw krb5kdc[22937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635762, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.121.189@KRBTEST.COM for oryx/127.16.121.189@KRBTEST.COM
I20260928 22:49:22.309829 23518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:22.303363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.189:46125 (local address 127.0.0.1:49832)
0928 22:49:22.303425 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:22.303436 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:22.303478 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:22.303593 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:22.303593 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:22.303706 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:22.303755 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:22.303756 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.304385 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.304386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:22.304917 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:22.304918 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.305305 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.305306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:22.305359 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:22.305779 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:22.305784 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:22.307148 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:22.308798 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:22.308799 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:22.308799 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:22.308966 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:22.309170 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:22.309170 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:22.309171 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:22.309223 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:22.309476 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:22.309477 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:22.309515 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 22:49:22.309583 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 22:49:22.309692 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 22:49:22.309837 23517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:22.303421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49832 (local address 127.16.121.189:46125)
0928 22:49:22.303461 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:22.303463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:22.303469 (+ 6us) server_negotiation.cc:407] Connection header received
0928 22:49:22.303536 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:22.303537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:22.303544 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:22.303589 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:22.303807 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.304313 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.305013 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.305218 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.305299 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:22.307247 (+ 1948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:22.307249 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:22.307249 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:22.307256 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:22.308692 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:22.309051 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:22.309052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:22.309053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:22.309105 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:22.309298 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:22.309298 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:22.309299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:22.309411 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:22.309487 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:22.309612 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 22:49:22.309761 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":87,"server-negotiator.queue_time_us":21}
I20260928 22:49:22.311772 23514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559: Bootstrap starting.
I20260928 22:49:22.312599 23514 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:22.312839 23514 log.cc:824] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559: Log is configured to *not* fsync() on all Append() calls
I20260928 22:49:22.313750 23514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559: No bootstrap required, opened a new log
I20260928 22:49:22.315481 23514 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } peers { permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 46125 } }
I20260928 22:49:22.315618 23514 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:22.315640 23514 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5fee57a42a14e1ab90bc3db2e453559, State: Initialized, Role: FOLLOWER
I20260928 22:49:22.315771 23514 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } peers { permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 46125 } }
I20260928 22:49:22.316061 23514 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:49:22.316049 23520 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } peers { permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 46125 } } }
I20260928 22:49:22.316164 23520 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 22:49:22.317996 23531 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:49:22.318074 23531 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:49:22.319947 23446 master_runner.cc:428] Detected that this master a5fee57a42a14e1ab90bc3db2e453559 is joining an existing cluster
I20260928 22:49:22.320025 23446 master_runner.cc:432] Initiating AddMaster RPC to add 127.16.121.189:46125
I20260928 22:49:22.321034 23520 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:49:22.321130 23520 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } peers { permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 46125 } }
I20260928 22:49:22.321379 23520 leader_election.cc:288] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c2ca4515ad9482a8d102db695d461ec (127.16.121.190:39331)
I20260928 22:49:22.321604 22992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a5fee57a42a14e1ab90bc3db2e453559" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c2ca4515ad9482a8d102db695d461ec" is_pre_election: true
I20260928 22:49:22.321996 23464 leader_election.cc:302] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: a5fee57a42a14e1ab90bc3db2e453559; no voters: 6c2ca4515ad9482a8d102db695d461ec
I20260928 22:49:22.322201 23520 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 22:49:22.328948 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:22.321104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46890 (local address 127.16.121.190:39331)
0928 22:49:22.321158 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:22.321160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:22.321382 (+ 222us) server_negotiation.cc:407] Connection header received
0928 22:49:22.321499 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:22.321501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:22.321515 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:22.321553 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:22.322067 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.322830 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.323835 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.324154 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.324197 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:22.325326 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:22.325328 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:22.325328 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:22.325336 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:22.327446 (+ 2110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:22.327968 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:22.327969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:22.327969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:22.328040 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:22.328271 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:22.328273 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:22.328273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:22.328447 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:22.328536 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:22.328701 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 22:49:22.328795 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 22:49:22.328961 23537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:22.321058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:39331 (local address 127.0.0.1:46890)
0928 22:49:22.321315 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:22.321327 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:22.321410 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:22.321629 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:22.321630 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:22.321871 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:22.321959 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:22.321962 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.322916 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.322917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:22.323726 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:22.323727 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.324293 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.324294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:22.324356 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:22.324885 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:22.324891 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:22.325225 (+ 334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:22.327548 (+ 2323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:22.327549 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:22.327550 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:22.327783 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:22.328118 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:22.328119 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:22.328119 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:22.328190 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:22.328525 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:22.328526 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:22.328581 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 22:49:22.328672 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 22:49:22.328844 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":601,"threads_started":1}
I20260928 22:49:22.330078 23517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:22.321108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49840 (local address 127.16.121.189:46125)
0928 22:49:22.321195 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:22.321196 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:22.322070 (+ 874us) server_negotiation.cc:407] Connection header received
0928 22:49:22.322148 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:22.322148 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:22.322157 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:22.322181 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:22.322680 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.323494 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.325046 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.325327 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.325424 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:22.326795 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:22.326797 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:22.326797 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:22.326805 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:22.328919 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:22.329301 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:22.329302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:22.329302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:22.329352 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:22.329521 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:22.329522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:22.329522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:22.329633 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:22.329673 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:22.329903 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 22:49:22.329986 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 22:49:22.330392 23538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:22.321154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.189:46125 (local address 127.0.0.1:49840)
0928 22:49:22.322014 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:22.322026 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:22.322092 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:22.322310 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:22.322311 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:22.322527 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:22.322589 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:22.322590 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.323608 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.323609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:22.324454 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:22.324456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.325422 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.325423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:22.325500 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:22.326144 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:22.326150 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:22.326668 (+ 518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:22.329009 (+ 2341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:22.329009 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:22.329010 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:22.329206 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:22.329414 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:22.329414 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:22.329415 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:22.329462 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:22.329711 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:22.329712 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:22.329746 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 22:49:22.330162 (+ 416us) client_negotiation.cc:240] Negotiation successful
0928 22:49:22.330295 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":832,"thread_start_us":105,"threads_started":1}
Sep 28 22:49:22 dist-test-slave-56vw krb5kdc[22937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635761, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.121.190@KRBTEST.COM for oryx/127.16.121.189@KRBTEST.COM
I20260928 22:49:22.339109 23517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:22.331713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49842 (local address 127.16.121.189:46125)
0928 22:49:22.331763 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:22.331765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:22.332017 (+ 252us) server_negotiation.cc:407] Connection header received
0928 22:49:22.332145 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:22.332146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:22.332157 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:22.332206 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:22.332690 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.333167 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.333960 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.334166 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.334250 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:22.336347 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:22.336348 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:22.336349 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:22.336356 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:22.337756 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:22.338191 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:22.338192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:22.338193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:22.338255 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:22.338475 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:22.338475 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:22.338476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:22.338599 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:22.338678 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:22.338870 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 22:49:22.338956 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 22:49:22.339124 23539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:22.331591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.189:46125 (local address 127.0.0.1:49842)
0928 22:49:22.331810 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:22.331923 (+ 113us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:22.332058 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:22.332211 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:22.332215 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:22.332542 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:22.332598 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:22.332601 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.333248 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.333250 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:22.333860 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:22.333862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.334251 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.334252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:22.334338 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:22.334843 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:22.334848 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:22.336253 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:22.337872 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:22.337879 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:22.337882 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:22.338099 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:22.338326 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:22.338327 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:22.338327 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:22.338395 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:22.338669 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:22.338677 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:22.338739 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 22:49:22.338843 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 22:49:22.338979 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":82,"threads_started":1}
I20260928 22:49:22.339859 22982 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.16.121.189:46125: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559"
member_type: NON_VOTER
last_known_addr {
host: "127.16.121.189"
port: 46125
}
attrs {
promote: true
}
}
dest_uuid: "6c2ca4515ad9482a8d102db695d461ec"
cas_config_opid_index: -1
I20260928 22:49:22.340204 22982 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } peers { permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: NON_VOTER last_known_addr { host: "127.16.121.189" port: 46125 } attrs { promote: true } }
I20260928 22:49:22.340698 23541 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER a5fee57a42a14e1ab90bc3db2e453559 (127.16.121.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } peers { permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: NON_VOTER last_known_addr { host: "127.16.121.189" port: 46125 } attrs { promote: true } } }
I20260928 22:49:22.341135 23488 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:22.341704 23541 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.16.121.189:46125
I20260928 22:49:22.341929 23540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6c2ca4515ad9482a8d102db695d461ec" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } peers { permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: NON_VOTER last_known_addr { host: "127.16.121.189" port: 46125 } attrs { promote: true } } }
I20260928 22:49:22.342065 23446 master.cc:537] Master@127.16.121.189:46125 shutting down...
I20260928 22:49:22.342089 23540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec [sys.catalog]: This master's current role is: LEADER
I20260928 22:49:22.342123 23488 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [term 1 FOLLOWER]: Refusing update from remote peer 6c2ca4515ad9482a8d102db695d461ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 22:49:22.342427 23540 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: NON_VOTER last_known_addr { host: "127.16.121.189" port: 46125 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 22:49:22.343266 22966 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 22:49:22.343734 22966 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec -> Peer a5fee57a42a14e1ab90bc3db2e453559 (127.16.121.189:46125): Couldn't send request to peer a5fee57a42a14e1ab90bc3db2e453559. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 22:49:22.345149 23446 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:49:22.345239 23446 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:49:22.345287 23446 tablet_replica.cc:330] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559: stopping tablet replica
I20260928 22:49:22.358186 23446 master.cc:552] Master@127.16.121.189:46125 shutdown complete.
I20260928 22:49:22.358240 23446 master_runner.cc:305] Clearing existing system tablet
I20260928 22:49:22.358381 23446 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:22.358457 23446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:22.359174 23446 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.002s sys 0.000s
I20260928 22:49:22.359687 23545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:22.359822 23446 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:22.359881 23446 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:22.359903 23446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-1/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-1/wal
uuid: "a5fee57a42a14e1ab90bc3db2e453559"
format_stamp: "Formatted at 2026-09-28 22:49:22 on dist-test-slave-56vw"
server_key: "68a8843f90e7ea3ef4a8391b02023070"
server_key_iv: "48c920f8fa6f70879dc5c926998dd7ba"
server_key_version: "encryptionkey@0"
I20260928 22:49:22.360549 23446 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:49:22.362033 23446 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 22:49:22.362094 23446 log.cc:1196] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 22:49:22.362314 23446 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559: Deleting consensus metadata
I20260928 22:49:22.362524 23446 master_runner.cc:315] Copying system tablet from 127.16.121.190:39331
I20260928 22:49:22.363227 23446 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559: tablet copy: Beginning tablet copy session from remote peer at address 127.16.121.190:39331
I20260928 22:49:22.369642 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:22.363656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46892 (local address 127.16.121.190:39331)
0928 22:49:22.363708 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:22.363710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:22.363774 (+ 64us) server_negotiation.cc:407] Connection header received
0928 22:49:22.363852 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:22.363853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:22.363864 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:22.363924 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:22.364295 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.364855 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.365613 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.365821 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.365909 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:22.366913 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:22.366914 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:22.366915 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:22.366923 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:22.368410 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:22.368794 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:22.368796 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:22.368796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:22.368865 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:22.369069 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:22.369070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:22.369070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:22.369199 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:22.369281 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:22.369456 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 22:49:22.369543 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 22:49:22.369656 23551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:22.363573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:39331 (local address 127.0.0.1:46892)
0928 22:49:22.363700 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:22.363714 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:22.363793 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:22.363931 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:22.363932 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:22.364135 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:22.364193 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:22.364195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.364969 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.364970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:22.365523 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:22.365525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.365911 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.365912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:22.365980 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:22.366504 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:22.366516 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:22.366814 (+ 298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:22.368518 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:22.368519 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:22.368519 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:22.368704 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:22.368933 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:22.368934 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:22.368934 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:22.368998 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:22.369269 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:22.369270 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:22.369323 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 22:49:22.369436 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 22:49:22.369544 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":77,"threads_started":1}
I20260928 22:49:22.370249 23002 tablet_copy_service.cc:140] P 6c2ca4515ad9482a8d102db695d461ec: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a5fee57a42a14e1ab90bc3db2e453559 ({username='oryx', principal='oryx/127.16.121.189@KRBTEST.COM'} at 127.0.0.1:46892)
I20260928 22:49:22.370345 23002 tablet_copy_service.cc:161] P 6c2ca4515ad9482a8d102db695d461ec: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a5fee57a42a14e1ab90bc3db2e453559 at {username='oryx', principal='oryx/127.16.121.189@KRBTEST.COM'} at 127.0.0.1:46892: session id = a5fee57a42a14e1ab90bc3db2e453559-00000000000000000000000000000000
I20260928 22:49:22.371265 23002 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 22:49:22.371955 23446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 22:49:22.372924 23446 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559: tablet copy: Starting download of 0 data blocks...
I20260928 22:49:22.373106 23446 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559: tablet copy: Starting download of 1 WAL segments...
I20260928 22:49:22.374205 23446 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 22:49:22.375303 23002 tablet_copy_service.cc:342] P 6c2ca4515ad9482a8d102db695d461ec: Request end of tablet copy session a5fee57a42a14e1ab90bc3db2e453559-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.16.121.189@KRBTEST.COM'} at 127.0.0.1:46892
I20260928 22:49:22.375393 23002 tablet_copy_service.cc:433] P 6c2ca4515ad9482a8d102db695d461ec: ending tablet copy session a5fee57a42a14e1ab90bc3db2e453559-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a5fee57a42a14e1ab90bc3db2e453559
I20260928 22:49:22.377887 23446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:22.379278 23552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:22.379302 23553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:22.379489 23446 server_base.cc:1034] running on GCE node
W20260928 22:49:22.379590 23555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:22.379772 23446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:22.379967 23446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:22.381114 23446 hybrid_clock.cc:648] HybridClock initialized: now 1790635762381101 us; error 30 us; skew 500 ppm
Sep 28 22:49:22 dist-test-slave-56vw krb5kdc[22937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635762, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.121.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:22.382632 23446 init.cc:377] Logged in from keytab as oryx/127.16.121.189@KRBTEST.COM (short username oryx)
I20260928 22:49:22.382952 23446 webserver.cc:460] Webserver started at http://127.16.121.189:45591/ using document root <none> and password file <none>
I20260928 22:49:22.383114 23446 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:22.383157 23446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:22.383845 23446 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:49:22.384444 23562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:22.384567 23446 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:49:22.384615 23446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-1/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-1/wal
uuid: "a5fee57a42a14e1ab90bc3db2e453559"
format_stamp: "Formatted at 2026-09-28 22:49:22 on dist-test-slave-56vw"
server_key: "68a8843f90e7ea3ef4a8391b02023070"
server_key_iv: "48c920f8fa6f70879dc5c926998dd7ba"
server_key_version: "encryptionkey@0"
I20260928 22:49:22.384684 23446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:22.409072 23446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:22.410768 23446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:22.414000 23446 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.189:46125
I20260928 22:49:22.414017 23614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.189:46125 every 8 connection(s)
I20260928 22:49:22.414418 23446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-1/data/info.pb
I20260928 22:49:22.415186 23615 sys_catalog.cc:263] Verifying existing consensus state
I20260928 22:49:22.415639 23615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559: Bootstrap starting.
I20260928 22:49:22.417940 23615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 22:49:22.418154 23615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559: Bootstrap complete.
I20260928 22:49:22.418362 23615 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } peers { permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: NON_VOTER last_known_addr { host: "127.16.121.189" port: 46125 } attrs { promote: true } }
I20260928 22:49:22.418443 23615 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a5fee57a42a14e1ab90bc3db2e453559, State: Initialized, Role: LEARNER
I20260928 22:49:22.418471 23615 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } peers { permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: NON_VOTER last_known_addr { host: "127.16.121.189" port: 46125 } attrs { promote: true } }
I20260928 22:49:22.418674 23615 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:49:22.418689 23618 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } peers { permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: NON_VOTER last_known_addr { host: "127.16.121.189" port: 46125 } attrs { promote: true } } }
I20260928 22:49:22.418805 23618 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [sys.catalog]: This master's current role is: LEARNER
I20260928 22:49:22.420209 23629 catalog_manager.cc:1259] Loaded cluster ID: f581d50fabd146eab37464bd687b1b1c
I20260928 22:49:22.420341 23629 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559: loading cluster ID for follower catalog manager: success
I20260928 22:49:22.421303 23629 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559: acquiring CA information for follower catalog manager: success
I20260928 22:49:22.421623 23629 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 22:49:22.462019 23630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:22.454759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49874 (local address 127.16.121.189:46125)
0928 22:49:22.454953 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:22.454956 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:22.454964 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:49:22.454999 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:22.455000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:22.455009 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:22.455140 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:22.455890 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.456620 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.457429 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.457641 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.457724 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:22.459330 (+ 1606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:22.459332 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:22.459332 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:22.459385 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:22.460844 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:22.461207 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:22.461208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:22.461209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:22.461270 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:22.461479 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:22.461480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:22.461480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:22.461598 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:22.461711 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:22.461815 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 22:49:22.461906 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":108,"threads_started":1}
WARNING: no policy specified for oryx/127.16.121.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.121.188@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.121.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.188.keytab.
Entry for principal oryx/127.16.121.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.188.keytab.
I20260928 22:49:22.491577 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.188:32847
--webserver_interface=127.16.121.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.188.keytab
--principal=oryx/127.16.121.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:33939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:39331,127.16.121.189:46125,127.16.121.188:32847
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:49:22.580964 23633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:22.581212 23633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:22.581238 23633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:22.582651 23633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:49:22.582721 23633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:22.582737 23633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:22.582751 23633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:49:22.582763 23633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:49:22.584553 23633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:33939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:39331,127.16.121.189:46125,127.16.121.188:32847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.188.keytab
--principal=oryx/127.16.121.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.188:32847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.121.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:22.584916 23633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:22.585191 23633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:22.590885 23638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:22.590889 23639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:22.590977 23633 server_base.cc:1034] running on GCE node
W20260928 22:49:22.590885 23641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:22.591406 23633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:22.591789 23633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:22.592970 23633 hybrid_clock.cc:648] HybridClock initialized: now 1790635762592957 us; error 38 us; skew 500 ppm
Sep 28 22:49:22 dist-test-slave-56vw krb5kdc[22937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635762, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.121.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:22.595934 23633 init.cc:377] Logged in from keytab as oryx/127.16.121.188@KRBTEST.COM (short username oryx)
I20260928 22:49:22.596437 23633 webserver.cc:460] Webserver started at http://127.16.121.188:37995/ using document root <none> and password file <none>
I20260928 22:49:22.596732 23633 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:22.596813 23633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:22.596912 23633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:22.597987 23633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-2/data/instance:
uuid: "f7b7f09129f5436ab464f41cbfd8c38f"
format_stamp: "Formatted at 2026-09-28 22:49:22 on dist-test-slave-56vw"
server_key: "ecd7c70bd62968a6f5f341abb5bad361"
server_key_iv: "366b86be1770c73b4bdefa2cd87e2449"
server_key_version: "encryptionkey@0"
I20260928 22:49:22.598307 23633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-2/wal/instance:
uuid: "f7b7f09129f5436ab464f41cbfd8c38f"
format_stamp: "Formatted at 2026-09-28 22:49:22 on dist-test-slave-56vw"
server_key: "ecd7c70bd62968a6f5f341abb5bad361"
server_key_iv: "366b86be1770c73b4bdefa2cd87e2449"
server_key_version: "encryptionkey@0"
I20260928 22:49:22.600003 23633 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 22:49:22.600934 23648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:22.601169 23633 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:22.601289 23633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-2/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-2/wal
uuid: "f7b7f09129f5436ab464f41cbfd8c38f"
format_stamp: "Formatted at 2026-09-28 22:49:22 on dist-test-slave-56vw"
server_key: "ecd7c70bd62968a6f5f341abb5bad361"
server_key_iv: "366b86be1770c73b4bdefa2cd87e2449"
server_key_version: "encryptionkey@0"
I20260928 22:49:22.601402 23633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:22.616130 23633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:22.618831 23633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:22.619109 23633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:22.623507 23633 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.188:32847
I20260928 22:49:22.623550 23700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.188:32847 every 8 connection(s)
I20260928 22:49:22.623925 23633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-2/data/info.pb
I20260928 22:49:22.624993 23701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:22.626668 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 23633
I20260928 22:49:22.626549 23701 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 22:49:22 dist-test-slave-56vw krb5kdc[22937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635762, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.121.188@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM
Sep 28 22:49:22 dist-test-slave-56vw krb5kdc[22937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.121.188@KRBTEST.COM
I20260928 22:49:22.637907 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:22.627494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46904 (local address 127.16.121.190:39331)
0928 22:49:22.627550 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:22.627553 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:22.627690 (+ 137us) server_negotiation.cc:407] Connection header received
0928 22:49:22.628241 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:22.628242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:22.628256 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:22.628332 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:22.628908 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.629417 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.630760 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.631162 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.631272 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:22.634009 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:22.634010 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:22.634010 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:22.634018 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:22.636013 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:22.636520 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:22.636522 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:22.636523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:22.636594 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:22.637005 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:22.637006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:22.637006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:22.637174 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:22.637259 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:22.637690 (+ 431us) server_negotiation.cc:299] Negotiation successful
0928 22:49:22.637801 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 22:49:22.638118 23703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:22.627338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:39331 (local address 127.0.0.1:46904)
0928 22:49:22.627570 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:22.627610 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:22.628103 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:22.628365 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:22.628367 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:22.628644 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:22.628812 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:22.628817 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.629547 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.629548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:22.630669 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:22.630671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.631418 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.631419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:22.631577 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:22.632258 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:22.632267 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:22.633903 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:22.636127 (+ 2224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:22.636133 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:22.636135 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:22.636427 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:22.636721 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:22.636723 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:22.636723 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:22.636916 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:22.637329 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:22.637337 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:22.637533 (+ 196us) client_negotiation.cc:769] Sending connection context
0928 22:49:22.637624 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 22:49:22.637859 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":115,"thread_start_us":91,"threads_started":1}
I20260928 22:49:22.638237 23704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:22.627338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33318 (local address 127.16.121.188:32847)
0928 22:49:22.627537 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:22.627548 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:22.627584 (+ 36us) server_negotiation.cc:407] Connection header received
0928 22:49:22.628103 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:22.628112 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:22.628138 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:22.628226 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:22.628942 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.630245 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.631344 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.631889 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.631962 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:22.634537 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:22.634545 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:22.634545 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:22.634552 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:22.636649 (+ 2097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:22.637143 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:22.637147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:22.637148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:22.637208 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:22.637509 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:22.637510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:22.637510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:22.637730 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:22.637904 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:22.638025 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 22:49:22.638106 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":144,"spinlock_wait_cycles":384,"thread_start_us":106,"threads_started":1}
I20260928 22:49:22.639096 23701 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.121.189" port: 46125 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 22:49:22 dist-test-slave-56vw krb5kdc[22937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635762, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.121.188@KRBTEST.COM for oryx/127.16.121.189@KRBTEST.COM
I20260928 22:49:22.646402 23703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:22.639381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.189:46125 (local address 127.0.0.1:49888)
0928 22:49:22.639563 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:22.639574 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:22.639614 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:22.639749 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:22.639750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:22.639877 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:22.639923 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:22.639924 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.640668 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.640669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:22.641245 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:22.641247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.641709 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.641710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:22.641769 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:22.642238 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:22.642243 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:22.643644 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:22.645412 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:22.645413 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:22.645414 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:22.645591 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:22.645807 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:22.645808 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:22.645809 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:22.645876 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:22.646124 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:22.646126 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:22.646187 (+ 61us) client_negotiation.cc:769] Sending connection context
0928 22:49:22.646253 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 22:49:22.646333 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162}
I20260928 22:49:22.646451 23630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:22.639510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49888 (local address 127.16.121.189:46125)
0928 22:49:22.639599 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:22.639601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:22.639612 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:49:22.639663 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:22.639664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:22.639677 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:22.639734 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:22.639992 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.640591 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.641414 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.641618 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.641687 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:22.643768 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:22.643769 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:22.643770 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:22.643777 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:22.645327 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:22.645661 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:22.645662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:22.645662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:22.645712 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:22.645940 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:22.645941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:22.645941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:22.646042 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:22.646092 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:22.646278 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 22:49:22.646345 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 22:49:22.646893 23701 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.121.188" port: 32847 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 22:49:22 dist-test-slave-56vw krb5kdc[22937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635762, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.121.188@KRBTEST.COM for oryx/127.16.121.188@KRBTEST.COM
I20260928 22:49:22.654495 23703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:22.647220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:32847 (local address 127.0.0.1:33322)
0928 22:49:22.647263 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:22.647274 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:22.647327 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:22.647452 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:22.647453 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:22.647589 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:22.647618 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:22.647619 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.648405 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.648406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:22.648889 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:22.648890 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.649262 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.649263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:22.649311 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:22.649728 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:22.649732 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:22.651186 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:22.653188 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:22.653190 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:22.653190 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:22.653400 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:22.653736 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:22.653737 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:22.653737 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:22.653808 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:22.654104 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:22.654105 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:22.654147 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 22:49:22.654244 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 22:49:22.654359 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 22:49:22.654500 23704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:22.647269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33322 (local address 127.16.121.188:32847)
0928 22:49:22.647303 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:22.647305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:22.647313 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:49:22.647386 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:22.647387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:22.647397 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:22.647449 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:22.647698 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.648237 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.648986 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.649182 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.649262 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:22.651469 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:22.651470 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:22.651471 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:22.651478 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:22.653087 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:22.653561 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:22.653562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:22.653562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:22.653627 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:22.653891 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:22.653892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:22.653893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:22.654030 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:22.654115 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:22.654291 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 22:49:22.654414 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":19}
I20260928 22:49:22.656296 23701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f: Bootstrap starting.
I20260928 22:49:22.657156 23701 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:22.657418 23701 log.cc:824] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f: Log is configured to *not* fsync() on all Append() calls
I20260928 22:49:22.658190 23701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f: No bootstrap required, opened a new log
I20260928 22:49:22.659742 23701 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } peers { permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 46125 } } peers { permanent_uuid: "f7b7f09129f5436ab464f41cbfd8c38f" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 32847 } }
I20260928 22:49:22.659891 23701 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:22.659915 23701 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7b7f09129f5436ab464f41cbfd8c38f, State: Initialized, Role: FOLLOWER
I20260928 22:49:22.660053 23701 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } peers { permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 46125 } } peers { permanent_uuid: "f7b7f09129f5436ab464f41cbfd8c38f" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 32847 } }
I20260928 22:49:22.660329 23706 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } peers { permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 46125 } } peers { permanent_uuid: "f7b7f09129f5436ab464f41cbfd8c38f" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 32847 } } }
I20260928 22:49:22.660408 23701 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:49:22.660419 23706 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f [sys.catalog]: This master's current role is: FOLLOWER
W20260928 22:49:22.662284 23717 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:49:22.662352 23717 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:49:22.665120 23633 master_runner.cc:428] Detected that this master f7b7f09129f5436ab464f41cbfd8c38f is joining an existing cluster
I20260928 22:49:22.665169 23633 master_runner.cc:432] Initiating AddMaster RPC to add 127.16.121.188:32847
I20260928 22:49:22.673800 23704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:22.666591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33330 (local address 127.16.121.188:32847)
0928 22:49:22.666631 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:22.666633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:22.666969 (+ 336us) server_negotiation.cc:407] Connection header received
0928 22:49:22.667072 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:22.667073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:22.667082 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:22.667107 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:22.667538 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.668277 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.669409 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.669689 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.669727 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:22.671303 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:22.671304 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:22.671305 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:22.671311 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:22.672599 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:22.672956 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:22.672957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:22.672957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:22.673014 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:22.673217 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:22.673218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:22.673219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:22.673334 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:22.673413 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:22.673561 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 22:49:22.673692 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:49:22.673882 23723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:22.666507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:32847 (local address 127.0.0.1:33330)
0928 22:49:22.666921 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:22.666930 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:22.666988 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:22.667162 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:22.667162 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:22.667403 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:22.667457 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:22.667459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.668372 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.668373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:22.669316 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:22.669319 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.669814 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.669815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:22.669870 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:22.670762 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:22.670766 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:22.671204 (+ 438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:22.672697 (+ 1493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:22.672698 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:22.672699 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:22.672872 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:22.673091 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:22.673092 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:22.673092 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:22.673146 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:22.673403 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:22.673404 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:22.673445 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 22:49:22.673539 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 22:49:22.673747 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":128,"thread_start_us":82,"threads_started":1}
I20260928 22:49:22.690450 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 23044
I20260928 22:49:22.693608 23630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:22.666680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49894 (local address 127.16.121.189:46125)
0928 22:49:22.666744 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:22.666746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:22.666757 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:49:22.666787 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:22.666788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:22.666801 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:22.666856 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:22.667490 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.668277 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.669485 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.669790 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.669829 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:22.688538 (+ 18709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:22.688541 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:22.688541 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:22.688552 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:22.690707 (+ 2155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:22.692543 (+ 1836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:22.692545 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:22.692546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:22.692624 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:22.692896 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:22.692897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:22.692898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:22.693061 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:22.693115 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:22.693383 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 22:49:22.693492 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 22:49:22.693634 23725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:22.666253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.189:46125 (local address 127.0.0.1:49894)
0928 22:49:22.666667 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:22.666678 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:22.666733 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:22.666913 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:22.666914 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:22.667300 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:22.667390 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:22.667392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.668372 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.668373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:22.669403 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:22.669405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.669900 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.669900 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:22.669957 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:22.688016 (+ 18059us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:22.688022 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:22.688396 (+ 374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:22.692138 (+ 3742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:22.692140 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:22.692140 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:22.692426 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:22.692749 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:22.692750 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:22.692750 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:22.692824 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:22.693161 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:22.693162 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:22.693218 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 22:49:22.693283 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 22:49:22.693440 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":158,"thread_start_us":100,"threads_started":1}
I20260928 22:49:22.696269 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.129:42243
--local_ip_for_outbound_sockets=127.16.121.129
--tserver_master_addrs=127.16.121.190:39331
--webserver_port=44831
--webserver_interface=127.16.121.129
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.129.keytab
--principal=oryx/127.16.121.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:33939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.121.190:39331,127.16.121.189:46125
--tserver_master_addrs=127.16.121.190:39331,127.16.121.189:46125,127.16.121.188:32847 with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:49:22.697233 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:22.666341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46910 (local address 127.16.121.190:39331)
0928 22:49:22.666391 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:22.666393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:22.666820 (+ 427us) server_negotiation.cc:407] Connection header received
0928 22:49:22.666885 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:22.666886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:22.666899 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:22.666949 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:22.667498 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.668287 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.669538 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.669854 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.669898 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:22.688581 (+ 18683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:22.688583 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:22.688584 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:22.688592 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:22.690758 (+ 2166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:22.696268 (+ 5510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:22.696270 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:22.696270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:22.696402 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:22.696631 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:22.696631 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:22.696632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:22.696798 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:22.696858 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:22.697055 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 22:49:22.697141 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 22:49:22.697252 23724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:22.666258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:39331 (local address 127.0.0.1:46910)
0928 22:49:22.666747 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:22.666757 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:22.666819 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:22.666995 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:22.666996 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:22.667296 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:22.667385 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:22.667388 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.668412 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.668413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:22.669430 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:22.669433 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.669987 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.669987 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:22.670389 (+ 402us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:22.687809 (+ 17420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:22.687817 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:22.688397 (+ 580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:22.695952 (+ 7555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:22.695953 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:22.695953 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:22.696158 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:22.696491 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:22.696491 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:22.696492 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:22.696564 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:22.696887 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:22.696888 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:22.696930 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 22:49:22.697002 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 22:49:22.697134 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":99,"thread_start_us":93,"threads_started":1}
I20260928 22:49:22.699312 23706 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:49:22.699427 23706 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } peers { permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 46125 } } peers { permanent_uuid: "f7b7f09129f5436ab464f41cbfd8c38f" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 32847 } }
I20260928 22:49:22.699900 23706 leader_election.cc:288] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c2ca4515ad9482a8d102db695d461ec (127.16.121.190:39331), a5fee57a42a14e1ab90bc3db2e453559 (127.16.121.189:46125)
I20260928 22:49:22.700438 23589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f7b7f09129f5436ab464f41cbfd8c38f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5fee57a42a14e1ab90bc3db2e453559" is_pre_election: true
I20260928 22:49:22.700569 23589 raft_consensus.cc:1799] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [term 1 LEARNER]: Handling vote request from an unknown peer f7b7f09129f5436ab464f41cbfd8c38f
I20260928 22:49:22.700628 23589 raft_consensus.cc:2396] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate f7b7f09129f5436ab464f41cbfd8c38f for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260928 22:49:22.701007 22992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f7b7f09129f5436ab464f41cbfd8c38f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c2ca4515ad9482a8d102db695d461ec" is_pre_election: true
I20260928 22:49:22.701216 23651 leader_election.cc:302] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7b7f09129f5436ab464f41cbfd8c38f; no voters: 6c2ca4515ad9482a8d102db695d461ec, a5fee57a42a14e1ab90bc3db2e453559
I20260928 22:49:22.701382 23706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:22.702416 23706 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
Sep 28 22:49:22 dist-test-slave-56vw krb5kdc[22937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635761, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.121.190@KRBTEST.COM for oryx/127.16.121.188@KRBTEST.COM
I20260928 22:49:22.709659 23539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:22.700207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:32847 (local address 127.0.0.1:33344)
0928 22:49:22.700282 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:22.700295 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:22.700347 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:22.700983 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:22.700984 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:22.701205 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:22.701264 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:22.701267 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.702323 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.702324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:22.703318 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:22.703321 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.703853 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.703854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:22.703941 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:22.704530 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:22.704536 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:22.706354 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:22.708485 (+ 2131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:22.708486 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:22.708486 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:22.708714 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:22.708951 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:22.708952 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:22.708952 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:22.709017 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:22.709310 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:22.709311 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:22.709363 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 22:49:22.709439 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 22:49:22.709552 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 22:49:22.709676 23704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:22.700723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33344 (local address 127.16.121.188:32847)
0928 22:49:22.700774 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:22.700776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:22.700785 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:49:22.700824 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:22.700825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:22.700837 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:22.700916 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:22.701384 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.702204 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.703412 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.703750 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.703834 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:22.706462 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:22.706463 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:22.706464 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:22.706472 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:22.708373 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:22.708799 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:22.708800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:22.708800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:22.708875 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:22.709086 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:22.709087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:22.709087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:22.709240 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:22.709314 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:22.709485 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 22:49:22.709573 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 22:49:22.710615 22982 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.16.121.188:32847: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f7b7f09129f5436ab464f41cbfd8c38f"
member_type: NON_VOTER
last_known_addr {
host: "127.16.121.188"
port: 32847
}
attrs {
promote: true
}
}
dest_uuid: "6c2ca4515ad9482a8d102db695d461ec"
cas_config_opid_index: 5
I20260928 22:49:22.710892 22982 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } peers { permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: NON_VOTER last_known_addr { host: "127.16.121.189" port: 46125 } attrs { promote: true } } peers { permanent_uuid: "f7b7f09129f5436ab464f41cbfd8c38f" member_type: NON_VOTER last_known_addr { host: "127.16.121.188" port: 32847 } attrs { promote: true } }
I20260928 22:49:22.711335 23541 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f7b7f09129f5436ab464f41cbfd8c38f (127.16.121.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } peers { permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: NON_VOTER last_known_addr { host: "127.16.121.189" port: 46125 } attrs { promote: true } } peers { permanent_uuid: "f7b7f09129f5436ab464f41cbfd8c38f" member_type: NON_VOTER last_known_addr { host: "127.16.121.188" port: 32847 } attrs { promote: true } } }
I20260928 22:49:22.712359 23541 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.16.121.188:32847
I20260928 22:49:22.712440 23540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6c2ca4515ad9482a8d102db695d461ec" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } peers { permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: NON_VOTER last_known_addr { host: "127.16.121.189" port: 46125 } attrs { promote: true } } peers { permanent_uuid: "f7b7f09129f5436ab464f41cbfd8c38f" member_type: NON_VOTER last_known_addr { host: "127.16.121.188" port: 32847 } attrs { promote: true } } }
I20260928 22:49:22.712524 23540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec [sys.catalog]: This master's current role is: LEADER
I20260928 22:49:22.712685 23675 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f [term 1 FOLLOWER]: Refusing update from remote peer 6c2ca4515ad9482a8d102db695d461ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 22:49:22.712953 23540 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7b7f09129f5436ab464f41cbfd8c38f" member_type: NON_VOTER last_known_addr { host: "127.16.121.188" port: 32847 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:49:22.713140 23633 master.cc:537] Master@127.16.121.188:32847 shutting down...
W20260928 22:49:22.713620 22966 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec -> Peer f7b7f09129f5436ab464f41cbfd8c38f (127.16.121.188:32847): Couldn't send request to peer f7b7f09129f5436ab464f41cbfd8c38f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 22:49:22.716907 23633 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:49:22.717000 23633 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:49:22.717039 23633 tablet_replica.cc:330] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f: stopping tablet replica
I20260928 22:49:22.720309 23539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:22.711398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.189:46125 (local address 127.0.0.1:49910)
0928 22:49:22.711575 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:22.711586 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:22.711621 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:22.715958 (+ 4337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:22.715959 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:22.715962 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:49:22.716016 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:22.716019 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.717318 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.717320 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:22.718697 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:22.718701 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.719972 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.719973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:22.720068 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:22.720102 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 22:49:22.720165 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 22:49:22.720218 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161}
I20260928 22:49:22.720790 23630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:22.712866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49910 (local address 127.16.121.189:46125)
0928 22:49:22.715578 (+ 2712us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:22.715581 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:22.715594 (+ 13us) server_negotiation.cc:407] Connection header received
0928 22:49:22.715624 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:22.715625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:22.715651 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:22.715733 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:49:22.716113 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.716956 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.718808 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.719753 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.719836 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:22.719931 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:22.720622 (+ 691us) server_negotiation.cc:299] Negotiation successful
0928 22:49:22.720697 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2681}
I20260928 22:49:22.721434 23589 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [term 1 LEARNER]: Refusing update from remote peer 6c2ca4515ad9482a8d102db695d461ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 22:49:22.721683 23540 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: NON_VOTER last_known_addr { host: "127.16.121.189" port: 46125 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 22:49:22.722370 23589 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f7b7f09129f5436ab464f41cbfd8c38f (127.16.121.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } peers { permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: NON_VOTER last_known_addr { host: "127.16.121.189" port: 46125 } attrs { promote: true } } peers { permanent_uuid: "f7b7f09129f5436ab464f41cbfd8c38f" member_type: NON_VOTER last_known_addr { host: "127.16.121.188" port: 32847 } attrs { promote: true } } }
I20260928 22:49:22.723676 23618 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c2ca4515ad9482a8d102db695d461ec. Latest consensus state: current_term: 1 leader_uuid: "6c2ca4515ad9482a8d102db695d461ec" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } peers { permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: NON_VOTER last_known_addr { host: "127.16.121.189" port: 46125 } attrs { promote: true } } peers { permanent_uuid: "f7b7f09129f5436ab464f41cbfd8c38f" member_type: NON_VOTER last_known_addr { host: "127.16.121.188" port: 32847 } attrs { promote: true } } }
I20260928 22:49:22.723763 23618 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [sys.catalog]: This master's current role is: LEARNER
I20260928 22:49:22.723799 23618 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c2ca4515ad9482a8d102db695d461ec" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } peers { permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: NON_VOTER last_known_addr { host: "127.16.121.189" port: 46125 } attrs { promote: true } } peers { permanent_uuid: "f7b7f09129f5436ab464f41cbfd8c38f" member_type: NON_VOTER last_known_addr { host: "127.16.121.188" port: 32847 } attrs { promote: true } } }
I20260928 22:49:22.723840 23618 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [sys.catalog]: This master's current role is: LEARNER
I20260928 22:49:22.724104 23540 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec: attempting to promote NON_VOTER a5fee57a42a14e1ab90bc3db2e453559 to VOTER
I20260928 22:49:22.724332 23540 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } peers { permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 46125 } attrs { promote: false } } peers { permanent_uuid: "f7b7f09129f5436ab464f41cbfd8c38f" member_type: NON_VOTER last_known_addr { host: "127.16.121.188" port: 32847 } attrs { promote: true } }
I20260928 22:49:22.724937 23589 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [term 1 LEARNER]: Refusing update from remote peer 6c2ca4515ad9482a8d102db695d461ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 22:49:22.725210 23729 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 46125 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 22:49:22.725287 23539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:22.724754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:32847 (local address 127.0.0.1:33360)
0928 22:49:22.725192 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:22.725251 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.188:32847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417}
I20260928 22:49:22.725699 23539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:22.725604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:32847 (local address 127.0.0.1:33362)
0928 22:49:22.725649 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:22.725673 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.188:32847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 22:49:22.725853 22966 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec -> Peer f7b7f09129f5436ab464f41cbfd8c38f (127.16.121.188:32847): Couldn't send request to peer f7b7f09129f5436ab464f41cbfd8c38f. Status: Network error: Client connection negotiation failed: client connection to 127.16.121.188:32847: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 22:49:22.726060 23541 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a5fee57a42a14e1ab90bc3db2e453559 (127.16.121.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } peers { permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 46125 } attrs { promote: false } } peers { permanent_uuid: "f7b7f09129f5436ab464f41cbfd8c38f" member_type: NON_VOTER last_known_addr { host: "127.16.121.188" port: 32847 } attrs { promote: true } } }
I20260928 22:49:22.726228 23589 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a5fee57a42a14e1ab90bc3db2e453559 (127.16.121.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } peers { permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 46125 } attrs { promote: false } } peers { permanent_uuid: "f7b7f09129f5436ab464f41cbfd8c38f" member_type: NON_VOTER last_known_addr { host: "127.16.121.188" port: 32847 } attrs { promote: true } } }
I20260928 22:49:22.727165 23541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6c2ca4515ad9482a8d102db695d461ec" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } peers { permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 46125 } attrs { promote: false } } peers { permanent_uuid: "f7b7f09129f5436ab464f41cbfd8c38f" member_type: NON_VOTER last_known_addr { host: "127.16.121.188" port: 32847 } attrs { promote: true } } }
I20260928 22:49:22.727250 23541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec [sys.catalog]: This master's current role is: LEADER
I20260928 22:49:22.727326 23541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6c2ca4515ad9482a8d102db695d461ec" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } peers { permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 46125 } attrs { promote: false } } peers { permanent_uuid: "f7b7f09129f5436ab464f41cbfd8c38f" member_type: NON_VOTER last_known_addr { host: "127.16.121.188" port: 32847 } attrs { promote: true } } }
I20260928 22:49:22.727375 23541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec [sys.catalog]: This master's current role is: LEADER
I20260928 22:49:22.727660 23728 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c2ca4515ad9482a8d102db695d461ec" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } peers { permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 46125 } attrs { promote: false } } peers { permanent_uuid: "f7b7f09129f5436ab464f41cbfd8c38f" member_type: NON_VOTER last_known_addr { host: "127.16.121.188" port: 32847 } attrs { promote: true } } }
I20260928 22:49:22.727790 23728 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:49:22.730281 23633 master.cc:552] Master@127.16.121.188:32847 shutdown complete.
I20260928 22:49:22.730330 23633 master_runner.cc:305] Clearing existing system tablet
I20260928 22:49:22.730489 23633 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:22.730540 23633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:22.731290 23633 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:49:22.731827 23733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:22.731966 23633 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:22.732016 23633 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:22.732038 23633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-2/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-2/wal
uuid: "f7b7f09129f5436ab464f41cbfd8c38f"
format_stamp: "Formatted at 2026-09-28 22:49:22 on dist-test-slave-56vw"
server_key: "ecd7c70bd62968a6f5f341abb5bad361"
server_key_iv: "366b86be1770c73b4bdefa2cd87e2449"
server_key_version: "encryptionkey@0"
I20260928 22:49:22.732806 23633 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:49:22.734411 23633 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 22:49:22.734501 23633 log.cc:1196] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 22:49:22.734706 23633 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f: Deleting consensus metadata
I20260928 22:49:22.734920 23633 master_runner.cc:315] Copying system tablet from 127.16.121.190:39331
I20260928 22:49:22.735581 23633 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f: tablet copy: Beginning tablet copy session from remote peer at address 127.16.121.190:39331
I20260928 22:49:22.746596 23739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:22.735901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:39331 (local address 127.0.0.1:46912)
0928 22:49:22.739218 (+ 3317us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:22.739243 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:22.739355 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:22.739591 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:22.739592 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:22.739812 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:22.739877 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:22.739879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.740868 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.740870 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:22.741721 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:22.741725 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.742236 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.742237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:22.742313 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:22.742868 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:22.742873 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:22.743320 (+ 447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:22.745243 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:22.745244 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:22.745244 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:22.745480 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:22.745748 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:22.745749 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:22.745750 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:22.745827 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:22.746179 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:22.746180 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:22.746297 (+ 117us) client_negotiation.cc:769] Sending connection context
0928 22:49:22.746375 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 22:49:22.746490 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3286,"thread_start_us":83,"threads_started":1}
I20260928 22:49:22.747092 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:22.735983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46912 (local address 127.16.121.190:39331)
0928 22:49:22.736034 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:22.736036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:22.739350 (+ 3314us) server_negotiation.cc:407] Connection header received
0928 22:49:22.739479 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:22.739481 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:22.739497 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:22.739576 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:22.739974 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.740770 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.741821 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.742148 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.742229 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:22.743421 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:22.743422 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:22.743423 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:22.743431 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:22.745146 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:22.745571 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:22.745573 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:22.745574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:22.745662 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:22.745903 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:22.745904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:22.745905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:22.746088 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:22.746175 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:22.746433 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 22:49:22.746540 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 22:49:22.747269 23002 tablet_copy_service.cc:140] P 6c2ca4515ad9482a8d102db695d461ec: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f7b7f09129f5436ab464f41cbfd8c38f ({username='oryx', principal='oryx/127.16.121.188@KRBTEST.COM'} at 127.0.0.1:46912)
I20260928 22:49:22.747319 23002 tablet_copy_service.cc:161] P 6c2ca4515ad9482a8d102db695d461ec: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f7b7f09129f5436ab464f41cbfd8c38f at {username='oryx', principal='oryx/127.16.121.188@KRBTEST.COM'} at 127.0.0.1:46912: session id = f7b7f09129f5436ab464f41cbfd8c38f-00000000000000000000000000000000
I20260928 22:49:22.747936 23002 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 22:49:22.748487 23633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 22:49:22.749845 23633 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f: tablet copy: Starting download of 0 data blocks...
I20260928 22:49:22.750005 23633 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f: tablet copy: Starting download of 1 WAL segments...
I20260928 22:49:22.750819 23633 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 22:49:22.751964 23002 tablet_copy_service.cc:342] P 6c2ca4515ad9482a8d102db695d461ec: Request end of tablet copy session f7b7f09129f5436ab464f41cbfd8c38f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.16.121.188@KRBTEST.COM'} at 127.0.0.1:46912
I20260928 22:49:22.752061 23002 tablet_copy_service.cc:433] P 6c2ca4515ad9482a8d102db695d461ec: ending tablet copy session f7b7f09129f5436ab464f41cbfd8c38f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f7b7f09129f5436ab464f41cbfd8c38f
I20260928 22:49:22.755924 23633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:22.757380 23743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:22.757380 23740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:22.757982 23633 server_base.cc:1034] running on GCE node
W20260928 22:49:22.759523 23741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:22.759724 23633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:22.759872 23633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:22.761018 23633 hybrid_clock.cc:648] HybridClock initialized: now 1790635762761015 us; error 30 us; skew 500 ppm
Sep 28 22:49:22 dist-test-slave-56vw krb5kdc[22937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635762, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.121.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:22.762704 23633 init.cc:377] Logged in from keytab as oryx/127.16.121.188@KRBTEST.COM (short username oryx)
I20260928 22:49:22.763000 23633 webserver.cc:460] Webserver started at http://127.16.121.188:41101/ using document root <none> and password file <none>
I20260928 22:49:22.763159 23633 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:22.763201 23633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:22.763993 23633 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:22.764719 23750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:22.764894 23633 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:22.764947 23633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-2/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-2/wal
uuid: "f7b7f09129f5436ab464f41cbfd8c38f"
format_stamp: "Formatted at 2026-09-28 22:49:22 on dist-test-slave-56vw"
server_key: "ecd7c70bd62968a6f5f341abb5bad361"
server_key_iv: "366b86be1770c73b4bdefa2cd87e2449"
server_key_version: "encryptionkey@0"
I20260928 22:49:22.765022 23633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:22.786809 23633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:22.789012 23633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:22.792331 23633 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.188:32847
I20260928 22:49:22.792743 23633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/master-2/data/info.pb
I20260928 22:49:22.793071 23802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.188:32847 every 8 connection(s)
I20260928 22:49:22.793754 23803 sys_catalog.cc:263] Verifying existing consensus state
I20260928 22:49:22.794324 23803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f: Bootstrap starting.
I20260928 22:49:22.797510 23803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 22:49:22.797737 23803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f: Bootstrap complete.
I20260928 22:49:22.798062 23803 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } peers { permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 46125 } attrs { promote: false } } peers { permanent_uuid: "f7b7f09129f5436ab464f41cbfd8c38f" member_type: NON_VOTER last_known_addr { host: "127.16.121.188" port: 32847 } attrs { promote: true } }
I20260928 22:49:22.798167 23803 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f7b7f09129f5436ab464f41cbfd8c38f, State: Initialized, Role: LEARNER
I20260928 22:49:22.798210 23803 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } peers { permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 46125 } attrs { promote: false } } peers { permanent_uuid: "f7b7f09129f5436ab464f41cbfd8c38f" member_type: NON_VOTER last_known_addr { host: "127.16.121.188" port: 32847 } attrs { promote: true } }
I20260928 22:49:22.798406 23803 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:49:22.798391 23806 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } peers { permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 46125 } attrs { promote: false } } peers { permanent_uuid: "f7b7f09129f5436ab464f41cbfd8c38f" member_type: NON_VOTER last_known_addr { host: "127.16.121.188" port: 32847 } attrs { promote: true } } }
I20260928 22:49:22.798703 23806 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f [sys.catalog]: This master's current role is: LEARNER
I20260928 22:49:22.800015 23817 catalog_manager.cc:1259] Loaded cluster ID: f581d50fabd146eab37464bd687b1b1c
I20260928 22:49:22.800101 23817 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f: loading cluster ID for follower catalog manager: success
I20260928 22:49:22.801636 23817 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f: acquiring CA information for follower catalog manager: success
I20260928 22:49:22.801998 23817 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 22:49:22.804925 23726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:22.805112 23726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:22.805137 23726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:22.806375 23726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:22.806414 23726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:22.806442 23726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.129
I20260928 22:49:22.808066 23726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:33939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.129.keytab
--principal=oryx/127.16.121.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.129:42243
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.121.129
--webserver_port=44831
--tserver_master_addrs=127.16.121.190:39331,127.16.121.189:46125,127.16.121.188:32847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.129
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:22.808331 23726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:22.808584 23726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:22.813748 23822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:22.813748 23823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:22.814105 23825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:22.814236 23726 server_base.cc:1034] running on GCE node
I20260928 22:49:22.814548 23726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:22.814824 23726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:22.815991 23726 hybrid_clock.cc:648] HybridClock initialized: now 1790635762815967 us; error 37 us; skew 500 ppm
Sep 28 22:49:22 dist-test-slave-56vw krb5kdc[22937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635762, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.121.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:22.818930 23726 init.cc:377] Logged in from keytab as oryx/127.16.121.129@KRBTEST.COM (short username oryx)
I20260928 22:49:22.819684 23726 webserver.cc:460] Webserver started at http://127.16.121.129:44831/ using document root <none> and password file <none>
I20260928 22:49:22.819988 23726 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:22.820082 23726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:22.822178 23726 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:49:22.823112 23832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:22.823292 23726 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:22.823375 23726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-0/wal
uuid: "fadc366fe3d44e068aee86188773c207"
format_stamp: "Formatted at 2026-09-28 22:49:21 on dist-test-slave-56vw"
server_key: "b04b8c9711778ac49b33251cbd0659d8"
server_key_iv: "5a6cf68701a29655d6dcb561f5a62738"
server_key_version: "encryptionkey@0"
I20260928 22:49:22.823765 23726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:22.838280 23726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:22.841388 23726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:22.841568 23726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:22.841847 23726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:49:22.842232 23726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:49:22.842294 23726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:22.842325 23726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:49:22.842350 23726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:22.848662 23726 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.129:42243
I20260928 22:49:22.848775 23945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.129:42243 every 8 connection(s)
I20260928 22:49:22.849128 23726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
I20260928 22:49:22.852946 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 23726
I20260928 22:49:22.853091 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 23178
Sep 28 22:49:22 dist-test-slave-56vw krb5kdc[22937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635762, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.121.129@KRBTEST.COM for oryx/127.16.121.188@KRBTEST.COM
Sep 28 22:49:22 dist-test-slave-56vw krb5kdc[22937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635762, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.121.129@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM
Sep 28 22:49:22 dist-test-slave-56vw krb5kdc[22937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635762, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.121.129@KRBTEST.COM for oryx/127.16.121.189@KRBTEST.COM
I20260928 22:49:22.860273 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.130:43775
--local_ip_for_outbound_sockets=127.16.121.130
--tserver_master_addrs=127.16.121.190:39331
--webserver_port=41013
--webserver_interface=127.16.121.130
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.130.keytab
--principal=oryx/127.16.121.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:33939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.121.190:39331,127.16.121.189:46125
--tserver_master_addrs=127.16.121.190:39331,127.16.121.189:46125,127.16.121.188:32847 with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:49:22.863619 23630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:22.850293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:33357 (local address 127.16.121.189:46125)
0928 22:49:22.850378 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:22.850381 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:22.850658 (+ 277us) server_negotiation.cc:407] Connection header received
0928 22:49:22.851354 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:22.851355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:22.851369 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:22.851422 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:22.853444 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.854326 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.857370 (+ 3044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.857613 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.857705 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:22.860033 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:22.860035 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:22.860035 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:22.860046 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:22.861908 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:22.862391 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:22.862392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:22.862392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:22.862459 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:22.862716 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:22.862717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:22.862717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:22.862897 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:22.862957 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:22.863368 (+ 411us) server_negotiation.cc:299] Negotiation successful
0928 22:49:22.863472 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 22:49:22.863647 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:22.850806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:40185 (local address 127.16.121.190:39331)
0928 22:49:22.850854 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:22.850856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:22.851298 (+ 442us) server_negotiation.cc:407] Connection header received
0928 22:49:22.851465 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:22.851466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:22.851481 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:22.851513 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:22.852508 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.853245 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.854896 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.855257 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.855328 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:22.859721 (+ 4393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:22.859723 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:22.859723 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:22.859733 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:22.862403 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:22.862796 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:22.862797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:22.862797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:22.862850 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:22.863082 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:22.863083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:22.863084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:22.863211 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:22.863259 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:22.863471 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 22:49:22.863559 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 22:49:22.863773 23951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:22.850520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:32847 (local address 127.16.121.129:56095)
0928 22:49:22.850780 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:22.850791 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:22.851229 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:22.852036 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:22.852037 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:22.852212 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:22.852388 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:22.852394 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.853394 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.853395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:22.854812 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:22.854813 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.855425 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.855427 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:22.855568 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:22.856392 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:22.856398 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:22.859587 (+ 3189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:22.861760 (+ 2173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:22.861770 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:22.861773 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:22.862220 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:22.862479 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:22.862480 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:22.862481 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:22.862632 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:22.862969 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:22.862976 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:22.863194 (+ 218us) client_negotiation.cc:769] Sending connection context
0928 22:49:22.863314 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 22:49:22.863521 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":10,"thread_start_us":81,"threads_started":1}
I20260928 22:49:22.863994 23953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:22.850721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:39331 (local address 127.16.121.129:40185)
0928 22:49:22.851175 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:22.851189 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:22.851282 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:22.851636 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:22.851640 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:22.852136 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:22.852388 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:22.852393 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.853733 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.853734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:22.854810 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:22.854813 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.855403 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.855404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:22.855569 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:22.856283 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:22.856294 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:22.859587 (+ 3293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:22.862486 (+ 2899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:22.862487 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:22.862487 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:22.862712 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:22.862927 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:22.862927 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:22.862928 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:22.862982 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:22.863321 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:22.863322 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:22.863357 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 22:49:22.863794 (+ 437us) client_negotiation.cc:240] Negotiation successful
0928 22:49:22.863899 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"mutex_wait_us":4,"thread_start_us":70,"threads_started":1}
I20260928 22:49:22.864104 23952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:22.850647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:56095 (local address 127.16.121.188:32847)
0928 22:49:22.850917 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:22.850919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:22.850936 (+ 17us) server_negotiation.cc:407] Connection header received
0928 22:49:22.851405 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:22.851407 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:22.851421 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:22.851492 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:22.852506 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.853276 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.854916 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.855246 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.855364 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:22.859761 (+ 4397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:22.859763 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:22.859763 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:22.859771 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:22.861655 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:22.862337 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:22.862339 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:22.862340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:22.862406 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:22.862713 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:22.862715 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:22.862715 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:22.862882 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:22.862944 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:22.863908 (+ 964us) server_negotiation.cc:299] Negotiation successful
0928 22:49:22.864000 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":57,"threads_started":1}
I20260928 22:49:22.864465 23947 heartbeater.cc:347] Connected to a master server at 127.16.121.190:39331
I20260928 22:49:22.864465 23946 heartbeater.cc:347] Connected to a master server at 127.16.121.188:32847
I20260928 22:49:22.864593 23947 heartbeater.cc:464] Registering TS with master...
I20260928 22:49:22.864593 23946 heartbeater.cc:464] Registering TS with master...
I20260928 22:49:22.864814 23946 heartbeater.cc:511] Master 127.16.121.188:32847 requested a full tablet report, sending...
I20260928 22:49:22.864814 23947 heartbeater.cc:511] Master 127.16.121.190:39331 requested a full tablet report, sending...
I20260928 22:49:22.865272 22982 ts_manager.cc:194] Re-registered known tserver with Master: fadc366fe3d44e068aee86188773c207 (127.16.121.129:42243)
I20260928 22:49:22.865406 23950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:22.850273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.189:46125 (local address 127.16.121.129:33357)
0928 22:49:22.850497 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:22.850566 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:22.851210 (+ 644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:22.851635 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:22.851639 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:22.852106 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:22.852388 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:22.852393 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.856439 (+ 4046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.856440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:22.857258 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:22.857261 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:22.857707 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:22.857708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:22.857781 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:22.858237 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:22.858269 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:22.859922 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:22.862015 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:22.862016 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:22.862016 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:22.862298 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:22.862530 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:22.862531 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:22.862531 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:22.862631 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:22.862999 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:22.863000 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:22.863195 (+ 195us) client_negotiation.cc:769] Sending connection context
0928 22:49:22.864693 (+ 1498us) client_negotiation.cc:240] Negotiation successful
0928 22:49:22.864988 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":82,"threads_started":1}
I20260928 22:49:22.865633 22982 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.121.129@KRBTEST.COM'} at 127.16.121.129:40185
I20260928 22:49:22.865957 23759 ts_manager.cc:194] Registered new tserver with Master: fadc366fe3d44e068aee86188773c207 (127.16.121.129:42243)
I20260928 22:49:22.866293 23948 heartbeater.cc:347] Connected to a master server at 127.16.121.189:46125
I20260928 22:49:22.866348 23948 heartbeater.cc:464] Registering TS with master...
I20260928 22:49:22.866505 23948 heartbeater.cc:511] Master 127.16.121.189:46125 requested a full tablet report, sending...
I20260928 22:49:22.866875 23579 ts_manager.cc:194] Registered new tserver with Master: fadc366fe3d44e068aee86188773c207 (127.16.121.129:42243)
W20260928 22:49:22.952178 23954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:22.952472 23954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:22.952518 23954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:22.954661 23954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:22.954754 23954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:22.954819 23954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.130
I20260928 22:49:22.956960 23954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:33939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.130.keytab
--principal=oryx/127.16.121.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.130:43775
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.121.130
--webserver_port=41013
--tserver_master_addrs=127.16.121.190:39331,127.16.121.189:46125,127.16.121.188:32847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.130
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:22.957290 23954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:22.957558 23954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:22.963316 23962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:22.963379 23959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:22.963326 23960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:22.963400 23954 server_base.cc:1034] running on GCE node
I20260928 22:49:22.963729 23954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:22.963986 23954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:22.965507 23954 hybrid_clock.cc:648] HybridClock initialized: now 1790635762965225 us; error 43 us; skew 500 ppm
Sep 28 22:49:22 dist-test-slave-56vw krb5kdc[22937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635762, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.121.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:22.968991 23954 init.cc:377] Logged in from keytab as oryx/127.16.121.130@KRBTEST.COM (short username oryx)
I20260928 22:49:22.969509 23954 webserver.cc:460] Webserver started at http://127.16.121.130:41013/ using document root <none> and password file <none>
I20260928 22:49:22.969758 23954 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:22.969825 23954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:22.971654 23954 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:22.972518 23969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:22.972733 23954 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:22.972841 23954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-1/wal
uuid: "c12e3d60fcc24875801961d9ddb3fe63"
format_stamp: "Formatted at 2026-09-28 22:49:21 on dist-test-slave-56vw"
server_key: "3bd5ca716ca34ecc9bcb2e9a28d8b0b1"
server_key_iv: "42d8106f1b34d39dfd47e3cd86ad8760"
server_key_version: "encryptionkey@0"
I20260928 22:49:22.973272 23954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:22.990413 23954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:22.993506 23954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:22.993808 23954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:22.994149 23954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:49:22.994556 23954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:49:22.994617 23954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:22.994663 23954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:49:22.994691 23954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:23.003384 23954 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.130:43775
I20260928 22:49:23.003469 24082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.130:43775 every 8 connection(s)
I20260928 22:49:23.003878 23954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
I20260928 22:49:23.007678 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 23954
I20260928 22:49:23.007840 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 23312
Sep 28 22:49:23 dist-test-slave-56vw krb5kdc[22937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635762, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.121.130@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM
Sep 28 22:49:23 dist-test-slave-56vw krb5kdc[22937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635762, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.121.130@KRBTEST.COM for oryx/127.16.121.189@KRBTEST.COM
Sep 28 22:49:23 dist-test-slave-56vw krb5kdc[22937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635762, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.121.130@KRBTEST.COM for oryx/127.16.121.188@KRBTEST.COM
I20260928 22:49:23.017103 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.131:36269
--local_ip_for_outbound_sockets=127.16.121.131
--tserver_master_addrs=127.16.121.190:39331
--webserver_port=45939
--webserver_interface=127.16.121.131
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.131.keytab
--principal=oryx/127.16.121.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:33939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.121.190:39331,127.16.121.189:46125
--tserver_master_addrs=127.16.121.190:39331,127.16.121.189:46125,127.16.121.188:32847 with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:49:23.018142 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:23.005382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:37625 (local address 127.16.121.190:39331)
0928 22:49:23.005433 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:23.005435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:23.005446 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:49:23.005782 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:23.005784 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:23.005797 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:23.005854 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:23.007154 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.008003 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.009277 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.009567 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.009607 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:23.012762 (+ 3155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:23.012764 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:23.012765 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:23.012774 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:23.015711 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:23.016383 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:23.016385 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:23.016385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:23.016469 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:23.016871 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:23.016873 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:23.016874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:23.017057 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:23.017108 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:23.017885 (+ 777us) server_negotiation.cc:299] Negotiation successful
0928 22:49:23.018002 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 22:49:23.018790 23952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:23.004789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:54097 (local address 127.16.121.188:32847)
0928 22:49:23.004976 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:23.004979 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:23.005228 (+ 249us) server_negotiation.cc:407] Connection header received
0928 22:49:23.005799 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:23.005801 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:23.005817 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:23.005849 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:23.006748 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.007353 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.009002 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.009373 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.009418 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:23.014086 (+ 4668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:23.014088 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:23.014089 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:23.014099 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:23.016951 (+ 2852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:23.017818 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:23.017820 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:23.017820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:23.017889 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:23.018118 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:23.018119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:23.018119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:23.018269 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:23.018315 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:23.018532 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 22:49:23.018656 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260928 22:49:23.018870 23630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:23.005453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:33053 (local address 127.16.121.189:46125)
0928 22:49:23.005518 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:23.005520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:23.005530 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:49:23.005800 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:23.005801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:23.005814 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:23.005849 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:23.006729 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.007430 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.009007 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.009315 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.009358 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:23.013511 (+ 4153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:23.013514 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:23.013514 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:23.013524 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:23.016086 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:23.017814 (+ 1728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:23.017816 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:23.017817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:23.017890 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:23.018132 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:23.018133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:23.018134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:23.018287 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:23.018345 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:23.018670 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 22:49:23.018767 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 22:49:23.019342 24088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:23.004773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:32847 (local address 127.16.121.130:54097)
0928 22:49:23.005134 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:23.005146 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:23.005698 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:23.005946 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:23.005951 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:23.006468 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:23.006628 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:23.006633 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.007470 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.007471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:23.008889 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:23.008892 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.009510 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.009511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:23.009602 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:23.010580 (+ 978us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:23.010586 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:23.013949 (+ 3363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:23.017074 (+ 3125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:23.017076 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:23.017076 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:23.017690 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:23.017962 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:23.017963 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:23.017963 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:23.018024 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:23.018362 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:23.018363 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:23.018410 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 22:49:23.018897 (+ 487us) client_negotiation.cc:240] Negotiation successful
0928 22:49:23.019023 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":37,"thread_start_us":127,"threads_started":1}
I20260928 22:49:23.019340 24089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:23.005215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.189:46125 (local address 127.16.121.130:33053)
0928 22:49:23.005316 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:23.005326 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:23.005694 (+ 368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:23.005946 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:23.005951 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:23.006403 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:23.006628 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:23.006633 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.007542 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.007543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:23.008886 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:23.008888 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.009450 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.009451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:23.009593 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:23.010305 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:23.010334 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:23.013371 (+ 3037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:23.017073 (+ 3702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:23.017074 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:23.017075 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:23.017693 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:23.017987 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:23.017988 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:23.017989 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:23.018053 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:23.018395 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:23.018396 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:23.018481 (+ 85us) client_negotiation.cc:769] Sending connection context
0928 22:49:23.018987 (+ 506us) client_negotiation.cc:240] Negotiation successful
0928 22:49:23.019165 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":24,"thread_start_us":60,"threads_started":1}
I20260928 22:49:23.019699 24087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:23.004771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:39331 (local address 127.16.121.130:37625)
0928 22:49:23.004940 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:23.004987 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:23.005674 (+ 687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:23.005950 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:23.005952 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:23.006396 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:23.006626 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:23.006631 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.008399 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.008401 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:23.009187 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:23.009189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.009685 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.009685 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:23.009771 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:23.010443 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:23.010447 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:23.012593 (+ 2146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:23.015833 (+ 3240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:23.015849 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:23.015853 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:23.016260 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:23.016566 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:23.016567 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:23.016567 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:23.016764 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:23.017161 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:23.017440 (+ 279us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:23.017668 (+ 228us) client_negotiation.cc:769] Sending connection context
0928 22:49:23.018603 (+ 935us) client_negotiation.cc:240] Negotiation successful
0928 22:49:23.018820 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":79,"threads_started":1}
I20260928 22:49:23.019886 24085 heartbeater.cc:347] Connected to a master server at 127.16.121.189:46125
I20260928 22:49:23.019979 24085 heartbeater.cc:464] Registering TS with master...
I20260928 22:49:23.020102 24084 heartbeater.cc:347] Connected to a master server at 127.16.121.190:39331
I20260928 22:49:23.020159 24084 heartbeater.cc:464] Registering TS with master...
I20260928 22:49:23.020200 24085 heartbeater.cc:511] Master 127.16.121.189:46125 requested a full tablet report, sending...
I20260928 22:49:23.020220 24084 heartbeater.cc:511] Master 127.16.121.190:39331 requested a full tablet report, sending...
I20260928 22:49:23.020639 22982 ts_manager.cc:194] Re-registered known tserver with Master: c12e3d60fcc24875801961d9ddb3fe63 (127.16.121.130:43775)
I20260928 22:49:23.021000 22982 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.121.130@KRBTEST.COM'} at 127.16.121.130:37625
I20260928 22:49:23.021260 23579 ts_manager.cc:194] Registered new tserver with Master: c12e3d60fcc24875801961d9ddb3fe63 (127.16.121.130:43775)
I20260928 22:49:23.022022 24083 heartbeater.cc:347] Connected to a master server at 127.16.121.188:32847
I20260928 22:49:23.022089 24083 heartbeater.cc:464] Registering TS with master...
I20260928 22:49:23.022163 24083 heartbeater.cc:511] Master 127.16.121.188:32847 requested a full tablet report, sending...
I20260928 22:49:23.022599 23759 ts_manager.cc:194] Registered new tserver with Master: c12e3d60fcc24875801961d9ddb3fe63 (127.16.121.130:43775)
W20260928 22:49:23.111168 24090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:23.111411 24090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:23.111439 24090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:23.112807 24090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:23.112867 24090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:23.112896 24090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.131
I20260928 22:49:23.114609 24090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:33939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.131.keytab
--principal=oryx/127.16.121.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.131:36269
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.121.131
--webserver_port=45939
--tserver_master_addrs=127.16.121.190:39331,127.16.121.189:46125,127.16.121.188:32847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.131
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:23.114915 24090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:23.115216 24090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:23.122368 24098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:23.122485 24096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:23.122402 24090 server_base.cc:1034] running on GCE node
W20260928 22:49:23.122725 24095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:23.123109 24090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:23.123461 24090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:23.124676 24090 hybrid_clock.cc:648] HybridClock initialized: now 1790635763124657 us; error 33 us; skew 500 ppm
Sep 28 22:49:23 dist-test-slave-56vw krb5kdc[22937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635763, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.121.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:23.128014 24090 init.cc:377] Logged in from keytab as oryx/127.16.121.131@KRBTEST.COM (short username oryx)
I20260928 22:49:23.128657 24090 webserver.cc:460] Webserver started at http://127.16.121.131:45939/ using document root <none> and password file <none>
I20260928 22:49:23.128919 24090 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:23.129016 24090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:23.130734 24090 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 22:49:23.131706 24105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:23.132015 24090 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 22:49:23.132115 24090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-2/wal
uuid: "e1ab1de193eb40a9bb803104ee08c680"
format_stamp: "Formatted at 2026-09-28 22:49:22 on dist-test-slave-56vw"
server_key: "649056e8767f8f0a232830196af6a4fb"
server_key_iv: "a0fcf1ad1605c858ea0717032d9711e9"
server_key_version: "encryptionkey@0"
I20260928 22:49:23.132486 24090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:23.145514 24090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:23.148245 24090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:23.148384 24090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:23.148597 24090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:49:23.148952 24090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:49:23.149013 24090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:23.149058 24090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:49:23.149085 24090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:23.156973 24090 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.131:36269
I20260928 22:49:23.157186 24218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.131:36269 every 8 connection(s)
I20260928 22:49:23.157510 24090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
Sep 28 22:49:23 dist-test-slave-56vw krb5kdc[22937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635763, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.121.131@KRBTEST.COM for oryx/127.16.121.188@KRBTEST.COM
I20260928 22:49:23.166512 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 24090
Sep 28 22:49:23 dist-test-slave-56vw krb5kdc[22937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635763, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.121.131@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM
Sep 28 22:49:23 dist-test-slave-56vw krb5kdc[22937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635763, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.121.131@KRBTEST.COM for oryx/127.16.121.189@KRBTEST.COM
I20260928 22:49:23.172474 24225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:23.158427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:39331 (local address 127.16.121.131:44803)
0928 22:49:23.159104 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:23.159116 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:23.159466 (+ 350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:23.159826 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:23.159827 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:23.160260 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:23.160537 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:23.160539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.161508 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.161509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:23.162693 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:23.162696 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.163241 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.163242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:23.163390 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:23.164232 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:23.164245 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:23.167758 (+ 3513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:23.170496 (+ 2738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:23.170509 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:23.170512 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:23.170915 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:23.171193 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:23.171194 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:23.171195 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:23.171363 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:23.171653 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:23.171660 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:23.171856 (+ 196us) client_negotiation.cc:769] Sending connection context
0928 22:49:23.171992 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 22:49:23.172197 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"mutex_wait_us":242,"thread_start_us":105,"threads_started":1}
I20260928 22:49:23.172964 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:23.158469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:44803 (local address 127.16.121.190:39331)
0928 22:49:23.158516 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:23.158519 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:23.159337 (+ 818us) server_negotiation.cc:407] Connection header received
0928 22:49:23.159610 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:23.159612 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:23.159627 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:23.159666 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:23.160694 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.161432 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.162843 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.163170 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.163237 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:23.167910 (+ 4673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:23.167912 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:23.167913 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:23.167923 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:23.170422 (+ 2499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:23.171018 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:23.171020 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:23.171020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:23.171120 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:23.171434 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:23.171434 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:23.171435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:23.171571 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:23.171626 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:23.172776 (+ 1150us) server_negotiation.cc:299] Negotiation successful
0928 22:49:23.172865 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 22:49:23.173328 24220 heartbeater.cc:347] Connected to a master server at 127.16.121.190:39331
I20260928 22:49:23.173449 24220 heartbeater.cc:464] Registering TS with master...
I20260928 22:49:23.173664 24220 heartbeater.cc:511] Master 127.16.121.190:39331 requested a full tablet report, sending...
I20260928 22:49:23.174132 22982 ts_manager.cc:194] Re-registered known tserver with Master: e1ab1de193eb40a9bb803104ee08c680 (127.16.121.131:36269)
I20260928 22:49:23.174634 22982 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.121.131@KRBTEST.COM'} at 127.16.121.131:44803
I20260928 22:49:23.175086 24223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:23.158893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.189:46125 (local address 127.16.121.131:55185)
0928 22:49:23.159511 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:23.159527 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:23.159955 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:23.160154 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:23.160155 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:23.160474 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:23.160537 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:23.160539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.161333 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.161334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:23.162693 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:23.162696 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.163244 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.163244 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:23.163390 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:23.164232 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:23.164245 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:23.167893 (+ 3648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:23.172873 (+ 4980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:23.172875 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:23.172875 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:23.173102 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:23.173467 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:23.173468 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:23.173468 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:23.173535 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:23.174434 (+ 899us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:23.174435 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:23.174487 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 22:49:23.174557 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 22:49:23.174706 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":46,"thread_start_us":70,"threads_started":1}
I20260928 22:49:23.176198 23630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:23.158991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:55185 (local address 127.16.121.189:46125)
0928 22:49:23.159072 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:23.159074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:23.159579 (+ 505us) server_negotiation.cc:407] Connection header received
0928 22:49:23.160054 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:23.160056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:23.160069 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:23.160134 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:23.160699 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.161217 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.162811 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.163041 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.163138 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:23.168006 (+ 4868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:23.168009 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:23.168009 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:23.168018 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:23.172620 (+ 4602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:23.173185 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:23.173186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:23.173186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:23.173251 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:23.173628 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:23.173629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:23.173629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:23.174327 (+ 698us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:23.174402 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:23.175444 (+ 1042us) server_negotiation.cc:299] Negotiation successful
0928 22:49:23.175535 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 22:49:23.176551 24221 heartbeater.cc:347] Connected to a master server at 127.16.121.189:46125
I20260928 22:49:23.176617 24221 heartbeater.cc:464] Registering TS with master...
I20260928 22:49:23.176718 24221 heartbeater.cc:511] Master 127.16.121.189:46125 requested a full tablet report, sending...
I20260928 22:49:23.177049 23579 ts_manager.cc:194] Registered new tserver with Master: e1ab1de193eb40a9bb803104ee08c680 (127.16.121.131:36269)
I20260928 22:49:23.178515 24224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:23.158427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:32847 (local address 127.16.121.131:44441)
0928 22:49:23.158611 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:23.158651 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:23.159455 (+ 804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:23.159816 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:23.159820 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:23.160262 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:23.160486 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:23.160492 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.161489 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.161490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:23.162618 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:23.162621 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.163217 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.163218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:23.163387 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:23.164243 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:23.164248 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:23.167718 (+ 3470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:23.177286 (+ 9568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:23.177287 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:23.177288 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:23.177506 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:23.177765 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:23.177765 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:23.177766 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:23.177826 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:23.178150 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:23.178151 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:23.178198 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 22:49:23.178266 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 22:49:23.178401 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":209,"spinlock_wait_cycles":768,"thread_start_us":104,"threads_started":1}
I20260928 22:49:23.178535 24232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:23.168994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49920 (local address 127.16.121.189:46125)
0928 22:49:23.169114 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:23.169117 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:23.169516 (+ 399us) server_negotiation.cc:407] Connection header received
0928 22:49:23.169670 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:23.169671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:23.169687 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:23.169732 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:23.170507 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.171400 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.172859 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.173182 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.173224 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:23.175308 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:23.175309 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:23.175309 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:23.175317 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:23.177137 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:23.177663 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:23.177664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:23.177665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:23.177718 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:23.177947 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:23.177948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:23.177948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:23.178065 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:23.178146 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:23.178355 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 22:49:23.178446 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":68,"threads_started":1}
I20260928 22:49:23.178572 23952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:23.158394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:44441 (local address 127.16.121.188:32847)
0928 22:49:23.158704 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:23.158707 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:23.158720 (+ 13us) server_negotiation.cc:407] Connection header received
0928 22:49:23.159594 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:23.159596 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:23.159609 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:23.159640 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:23.160585 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.161379 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.162730 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.163070 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.163139 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:23.167892 (+ 4753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:23.167894 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:23.167895 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:23.167905 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:23.171849 (+ 3944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:23.177612 (+ 5763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:23.177614 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:23.177614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:23.177685 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:23.177918 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:23.177919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:23.177920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:23.178059 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:23.178115 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:23.178337 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 22:49:23.178452 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276}
I20260928 22:49:23.179025 24219 heartbeater.cc:347] Connected to a master server at 127.16.121.188:32847
I20260928 22:49:23.179107 24219 heartbeater.cc:464] Registering TS with master...
I20260928 22:49:23.179162 24219 heartbeater.cc:511] Master 127.16.121.188:32847 requested a full tablet report, sending...
I20260928 22:49:23.179462 23759 ts_manager.cc:194] Registered new tserver with Master: e1ab1de193eb40a9bb803104ee08c680 (127.16.121.131:36269)
I20260928 22:49:23.181017 24230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:23.168109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46926 (local address 127.16.121.190:39331)
0928 22:49:23.168365 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:23.168367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:23.169127 (+ 760us) server_negotiation.cc:407] Connection header received
0928 22:49:23.169259 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:23.169260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:23.169276 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:23.169347 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:23.170444 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.174142 (+ 3698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.175130 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.175422 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.175466 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:23.177058 (+ 1592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:23.177059 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:23.177060 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:23.177068 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:23.179146 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:23.179751 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:23.179752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:23.179753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:23.179822 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:23.180201 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:23.180202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:23.180203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:23.180373 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:23.180491 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:23.180707 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 22:49:23.180798 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":104,"threads_started":1}
I20260928 22:49:23.182287 24234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:23.169385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33370 (local address 127.16.121.188:32847)
0928 22:49:23.169809 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:23.169811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:23.169982 (+ 171us) server_negotiation.cc:407] Connection header received
0928 22:49:23.170009 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:23.170010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:23.170025 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:23.170085 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:23.170830 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.171629 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.172749 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.173060 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.173128 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:23.177685 (+ 4557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:23.177687 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:23.177687 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:23.177695 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:23.180623 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:23.181334 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:23.181336 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:23.181345 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:23.181422 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:23.181678 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:23.181679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:23.181680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:23.181806 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:23.181935 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:23.182068 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 22:49:23.182160 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":217,"threads_started":1}
I20260928 22:49:23.183032 22982 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46926:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:49:23.184957 22982 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:49:23.193845 24242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:23.189895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40900 (local address 127.16.121.131:36269)
0928 22:49:23.190497 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:23.190500 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:23.190509 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:49:23.190546 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:23.190549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:23.190563 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:23.190638 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:49:23.190726 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.191718 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.192905 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.193460 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.193587 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:23.193657 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:23.193714 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 22:49:23.193753 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":110,"threads_started":1}
I20260928 22:49:23.193837 24241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:23.189722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.131:36269 (local address 127.0.0.1:40900)
0928 22:49:23.190032 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:23.190042 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:23.190078 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:23.190634 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:23.190635 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:23.190636 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:49:23.190669 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:23.190669 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.191874 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.191875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:23.192804 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:23.192807 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.193570 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.193571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:23.193632 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:23.193638 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 22:49:23.193694 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 22:49:23.193740 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":22,"thread_start_us":93,"threads_started":1}
I20260928 22:49:23.194360 23539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:23.189709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.130:43775 (local address 127.0.0.1:42820)
0928 22:49:23.189937 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:23.189948 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:23.189985 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:23.190459 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:23.190460 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:23.190462 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:49:23.190518 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:23.190520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.191873 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.191874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:23.193081 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:23.193084 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.194109 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.194110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:23.194183 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:23.194186 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:49:23.194238 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 22:49:23.194280 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":2,"thread_start_us":108,"threads_started":1}
I20260928 22:49:23.194545 24244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:23.190279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41398 (local address 127.16.121.129:42243)
0928 22:49:23.190440 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:23.190445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:23.190457 (+ 12us) server_negotiation.cc:407] Connection header received
0928 22:49:23.190502 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:23.190504 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:23.190523 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:23.190626 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:49:23.190843 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.191921 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.193226 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.193973 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.194068 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:23.194134 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:23.194406 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 22:49:23.194461 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":92,"threads_started":1}
I20260928 22:49:23.194582 24243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:23.190087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42820 (local address 127.16.121.130:43775)
0928 22:49:23.190278 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:23.190282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:23.190291 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:49:23.190332 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:23.190334 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:23.190351 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:23.190465 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:49:23.190576 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.191760 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.193202 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.193973 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.194104 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:23.194157 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:23.194251 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 22:49:23.194304 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":102,"threads_started":1}
I20260928 22:49:23.195101 24153 tablet_service.cc:1467] Processing CreateTablet for tablet 6d157dc401c7421cb8106ec5723d507b (DEFAULT_TABLE table=test-table [id=6e90f994a896415caef6552dedc54af8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:49:23.195281 24240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:23.189608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:42243 (local address 127.0.0.1:41398)
0928 22:49:23.189873 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:23.189886 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:23.189945 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:23.190700 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:23.190701 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:23.190704 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:49:23.190755 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:23.190757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.192006 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.192006 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:23.193128 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:23.193130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.194170 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.194171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:23.194239 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:23.194244 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:49:23.194294 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 22:49:23.194342 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":4}
I20260928 22:49:23.195539 24153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d157dc401c7421cb8106ec5723d507b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:23.195639 24017 tablet_service.cc:1467] Processing CreateTablet for tablet 6d157dc401c7421cb8106ec5723d507b (DEFAULT_TABLE table=test-table [id=6e90f994a896415caef6552dedc54af8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:49:23.196043 24017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d157dc401c7421cb8106ec5723d507b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:23.196394 23880 tablet_service.cc:1467] Processing CreateTablet for tablet 6d157dc401c7421cb8106ec5723d507b (DEFAULT_TABLE table=test-table [id=6e90f994a896415caef6552dedc54af8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:49:23.196718 23880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d157dc401c7421cb8106ec5723d507b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:23.199800 24245 tablet_bootstrap.cc:492] T 6d157dc401c7421cb8106ec5723d507b P c12e3d60fcc24875801961d9ddb3fe63: Bootstrap starting.
I20260928 22:49:23.199858 24246 tablet_bootstrap.cc:492] T 6d157dc401c7421cb8106ec5723d507b P e1ab1de193eb40a9bb803104ee08c680: Bootstrap starting.
I20260928 22:49:23.200482 24245 tablet_bootstrap.cc:654] T 6d157dc401c7421cb8106ec5723d507b P c12e3d60fcc24875801961d9ddb3fe63: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:23.200569 24246 tablet_bootstrap.cc:654] T 6d157dc401c7421cb8106ec5723d507b P e1ab1de193eb40a9bb803104ee08c680: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:23.200809 24245 log.cc:824] T 6d157dc401c7421cb8106ec5723d507b P c12e3d60fcc24875801961d9ddb3fe63: Log is configured to *not* fsync() on all Append() calls
I20260928 22:49:23.200902 24246 log.cc:824] T 6d157dc401c7421cb8106ec5723d507b P e1ab1de193eb40a9bb803104ee08c680: Log is configured to *not* fsync() on all Append() calls
I20260928 22:49:23.201444 24247 tablet_bootstrap.cc:492] T 6d157dc401c7421cb8106ec5723d507b P fadc366fe3d44e068aee86188773c207: Bootstrap starting.
I20260928 22:49:23.201656 24246 tablet_bootstrap.cc:492] T 6d157dc401c7421cb8106ec5723d507b P e1ab1de193eb40a9bb803104ee08c680: No bootstrap required, opened a new log
I20260928 22:49:23.201656 24245 tablet_bootstrap.cc:492] T 6d157dc401c7421cb8106ec5723d507b P c12e3d60fcc24875801961d9ddb3fe63: No bootstrap required, opened a new log
I20260928 22:49:23.201731 24245 ts_tablet_manager.cc:1397] T 6d157dc401c7421cb8106ec5723d507b P c12e3d60fcc24875801961d9ddb3fe63: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:49:23.201731 24246 ts_tablet_manager.cc:1397] T 6d157dc401c7421cb8106ec5723d507b P e1ab1de193eb40a9bb803104ee08c680: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:49:23.202129 24247 tablet_bootstrap.cc:654] T 6d157dc401c7421cb8106ec5723d507b P fadc366fe3d44e068aee86188773c207: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:23.202455 24247 log.cc:824] T 6d157dc401c7421cb8106ec5723d507b P fadc366fe3d44e068aee86188773c207: Log is configured to *not* fsync() on all Append() calls
I20260928 22:49:23.203557 24247 tablet_bootstrap.cc:492] T 6d157dc401c7421cb8106ec5723d507b P fadc366fe3d44e068aee86188773c207: No bootstrap required, opened a new log
I20260928 22:49:23.203641 24247 ts_tablet_manager.cc:1397] T 6d157dc401c7421cb8106ec5723d507b P fadc366fe3d44e068aee86188773c207: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:49:23.204071 24245 raft_consensus.cc:348] T 6d157dc401c7421cb8106ec5723d507b P c12e3d60fcc24875801961d9ddb3fe63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fadc366fe3d44e068aee86188773c207" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 42243 } } peers { permanent_uuid: "e1ab1de193eb40a9bb803104ee08c680" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36269 } } peers { permanent_uuid: "c12e3d60fcc24875801961d9ddb3fe63" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 43775 } }
I20260928 22:49:23.204268 24245 raft_consensus.cc:374] T 6d157dc401c7421cb8106ec5723d507b P c12e3d60fcc24875801961d9ddb3fe63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:23.204320 24245 raft_consensus.cc:729] T 6d157dc401c7421cb8106ec5723d507b P c12e3d60fcc24875801961d9ddb3fe63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c12e3d60fcc24875801961d9ddb3fe63, State: Initialized, Role: FOLLOWER
I20260928 22:49:23.204479 24245 consensus_queue.cc:260] T 6d157dc401c7421cb8106ec5723d507b P c12e3d60fcc24875801961d9ddb3fe63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fadc366fe3d44e068aee86188773c207" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 42243 } } peers { permanent_uuid: "e1ab1de193eb40a9bb803104ee08c680" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36269 } } peers { permanent_uuid: "c12e3d60fcc24875801961d9ddb3fe63" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 43775 } }
I20260928 22:49:23.204581 24246 raft_consensus.cc:348] T 6d157dc401c7421cb8106ec5723d507b P e1ab1de193eb40a9bb803104ee08c680 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fadc366fe3d44e068aee86188773c207" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 42243 } } peers { permanent_uuid: "e1ab1de193eb40a9bb803104ee08c680" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36269 } } peers { permanent_uuid: "c12e3d60fcc24875801961d9ddb3fe63" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 43775 } }
I20260928 22:49:23.204794 24245 ts_tablet_manager.cc:1428] T 6d157dc401c7421cb8106ec5723d507b P c12e3d60fcc24875801961d9ddb3fe63: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 22:49:23.204787 24246 raft_consensus.cc:374] T 6d157dc401c7421cb8106ec5723d507b P e1ab1de193eb40a9bb803104ee08c680 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:23.204845 24246 raft_consensus.cc:729] T 6d157dc401c7421cb8106ec5723d507b P e1ab1de193eb40a9bb803104ee08c680 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1ab1de193eb40a9bb803104ee08c680, State: Initialized, Role: FOLLOWER
I20260928 22:49:23.204963 24246 consensus_queue.cc:260] T 6d157dc401c7421cb8106ec5723d507b P e1ab1de193eb40a9bb803104ee08c680 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fadc366fe3d44e068aee86188773c207" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 42243 } } peers { permanent_uuid: "e1ab1de193eb40a9bb803104ee08c680" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36269 } } peers { permanent_uuid: "c12e3d60fcc24875801961d9ddb3fe63" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 43775 } }
I20260928 22:49:23.205221 24246 ts_tablet_manager.cc:1428] T 6d157dc401c7421cb8106ec5723d507b P e1ab1de193eb40a9bb803104ee08c680: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 22:49:23.205402 24220 heartbeater.cc:503] Master 127.16.121.190:39331 was elected leader, sending a full tablet report...
I20260928 22:49:23.205618 24084 heartbeater.cc:503] Master 127.16.121.190:39331 was elected leader, sending a full tablet report...
I20260928 22:49:23.208230 24247 raft_consensus.cc:348] T 6d157dc401c7421cb8106ec5723d507b P fadc366fe3d44e068aee86188773c207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fadc366fe3d44e068aee86188773c207" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 42243 } } peers { permanent_uuid: "e1ab1de193eb40a9bb803104ee08c680" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36269 } } peers { permanent_uuid: "c12e3d60fcc24875801961d9ddb3fe63" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 43775 } }
I20260928 22:49:23.208415 24247 raft_consensus.cc:374] T 6d157dc401c7421cb8106ec5723d507b P fadc366fe3d44e068aee86188773c207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:23.208459 24247 raft_consensus.cc:729] T 6d157dc401c7421cb8106ec5723d507b P fadc366fe3d44e068aee86188773c207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fadc366fe3d44e068aee86188773c207, State: Initialized, Role: FOLLOWER
I20260928 22:49:23.208590 24247 consensus_queue.cc:260] T 6d157dc401c7421cb8106ec5723d507b P fadc366fe3d44e068aee86188773c207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fadc366fe3d44e068aee86188773c207" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 42243 } } peers { permanent_uuid: "e1ab1de193eb40a9bb803104ee08c680" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36269 } } peers { permanent_uuid: "c12e3d60fcc24875801961d9ddb3fe63" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 43775 } }
I20260928 22:49:23.208868 24247 ts_tablet_manager.cc:1428] T 6d157dc401c7421cb8106ec5723d507b P fadc366fe3d44e068aee86188773c207: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260928 22:49:23.208990 23947 heartbeater.cc:503] Master 127.16.121.190:39331 was elected leader, sending a full tablet report...
I20260928 22:49:23.247965 24240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:23.244772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:32847 (local address 127.0.0.1:33380)
0928 22:49:23.244874 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:23.244889 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:23.244978 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:23.245225 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:23.245234 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:23.245237 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:49:23.245291 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:23.245293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.246041 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.246042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:23.246904 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:23.246909 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.247721 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.247722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:23.247782 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:23.247787 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:49:23.247850 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 22:49:23.247898 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 22:49:23.247964 24234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:23.244940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33380 (local address 127.16.121.188:32847)
0928 22:49:23.245031 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:23.245033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:23.245043 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:49:23.245074 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:23.245075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:23.245104 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:23.245176 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:49:23.245395 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.245922 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.247019 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.247620 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.247692 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:23.247766 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:23.247865 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 22:49:23.247907 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 22:49:23.248420 23777 raft_consensus.cc:1203] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260928 22:49:23.249439 23806 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c2ca4515ad9482a8d102db695d461ec. Latest consensus state: current_term: 1 leader_uuid: "6c2ca4515ad9482a8d102db695d461ec" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } peers { permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 46125 } attrs { promote: false } } peers { permanent_uuid: "f7b7f09129f5436ab464f41cbfd8c38f" member_type: NON_VOTER last_known_addr { host: "127.16.121.188" port: 32847 } attrs { promote: true } } }
I20260928 22:49:23.249557 23806 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f [sys.catalog]: This master's current role is: LEARNER
W20260928 22:49:23.255076 24086 tablet.cc:2369] T 6d157dc401c7421cb8106ec5723d507b P c12e3d60fcc24875801961d9ddb3fe63: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:49:23.298928 24252 raft_consensus.cc:482] T 6d157dc401c7421cb8106ec5723d507b P e1ab1de193eb40a9bb803104ee08c680 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:49:23.299151 24252 raft_consensus.cc:504] T 6d157dc401c7421cb8106ec5723d507b P e1ab1de193eb40a9bb803104ee08c680 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fadc366fe3d44e068aee86188773c207" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 42243 } } peers { permanent_uuid: "e1ab1de193eb40a9bb803104ee08c680" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36269 } } peers { permanent_uuid: "c12e3d60fcc24875801961d9ddb3fe63" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 43775 } }
I20260928 22:49:23.299649 24252 leader_election.cc:288] T 6d157dc401c7421cb8106ec5723d507b P e1ab1de193eb40a9bb803104ee08c680 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fadc366fe3d44e068aee86188773c207 (127.16.121.129:42243), c12e3d60fcc24875801961d9ddb3fe63 (127.16.121.130:43775)
I20260928 22:49:23.303813 24243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:23.300288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:32947 (local address 127.16.121.130:43775)
0928 22:49:23.300353 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:23.300355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:23.300366 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:49:23.300463 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:23.300464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:23.300473 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:23.300597 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:49:23.300980 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.301618 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.302833 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.303348 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.303466 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:23.303479 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:23.303668 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 22:49:23.303725 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 22:49:23.303830 24244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:23.300167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:51521 (local address 127.16.121.129:42243)
0928 22:49:23.300247 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:23.300250 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:23.300258 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:49:23.300291 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:23.300293 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:23.300304 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:23.300371 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:49:23.300688 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.301366 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.302864 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.303354 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.303520 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:23.303530 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:23.303707 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 22:49:23.303755 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 22:49:23.303833 24223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:23.300166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.130:43775 (local address 127.16.121.131:32947)
0928 22:49:23.300229 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:23.300239 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:23.300390 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:23.300683 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:23.300684 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:23.300686 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:49:23.300758 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:23.300760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.301892 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.301893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:23.302711 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:23.302713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.303487 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.303488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:23.303583 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:23.303586 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:49:23.303646 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 22:49:23.303721 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 22:49:23.303855 24224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:23.300021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:42243 (local address 127.16.121.131:51521)
0928 22:49:23.300070 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:23.300085 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:23.300152 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:23.300437 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:23.300439 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:23.300441 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:49:23.300535 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:23.300538 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.301505 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.301507 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:23.302719 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:23.302722 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.303527 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.303532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:23.303602 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:23.303606 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:49:23.303689 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 22:49:23.303731 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 22:49:23.304352 23900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d157dc401c7421cb8106ec5723d507b" candidate_uuid: "e1ab1de193eb40a9bb803104ee08c680" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fadc366fe3d44e068aee86188773c207" is_pre_election: true
I20260928 22:49:23.304358 24037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d157dc401c7421cb8106ec5723d507b" candidate_uuid: "e1ab1de193eb40a9bb803104ee08c680" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12e3d60fcc24875801961d9ddb3fe63" is_pre_election: true
I20260928 22:49:23.304549 24037 raft_consensus.cc:2454] T 6d157dc401c7421cb8106ec5723d507b P c12e3d60fcc24875801961d9ddb3fe63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1ab1de193eb40a9bb803104ee08c680 in term 0.
I20260928 22:49:23.304549 23900 raft_consensus.cc:2454] T 6d157dc401c7421cb8106ec5723d507b P fadc366fe3d44e068aee86188773c207 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1ab1de193eb40a9bb803104ee08c680 in term 0.
I20260928 22:49:23.304903 24109 leader_election.cc:302] T 6d157dc401c7421cb8106ec5723d507b P e1ab1de193eb40a9bb803104ee08c680 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e1ab1de193eb40a9bb803104ee08c680, fadc366fe3d44e068aee86188773c207; no voters:
I20260928 22:49:23.305047 24252 raft_consensus.cc:2781] T 6d157dc401c7421cb8106ec5723d507b P e1ab1de193eb40a9bb803104ee08c680 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:49:23.305097 24252 raft_consensus.cc:482] T 6d157dc401c7421cb8106ec5723d507b P e1ab1de193eb40a9bb803104ee08c680 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:49:23.305122 24252 raft_consensus.cc:3037] T 6d157dc401c7421cb8106ec5723d507b P e1ab1de193eb40a9bb803104ee08c680 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:23.305948 24252 raft_consensus.cc:504] T 6d157dc401c7421cb8106ec5723d507b P e1ab1de193eb40a9bb803104ee08c680 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fadc366fe3d44e068aee86188773c207" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 42243 } } peers { permanent_uuid: "e1ab1de193eb40a9bb803104ee08c680" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36269 } } peers { permanent_uuid: "c12e3d60fcc24875801961d9ddb3fe63" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 43775 } }
I20260928 22:49:23.306123 24252 leader_election.cc:288] T 6d157dc401c7421cb8106ec5723d507b P e1ab1de193eb40a9bb803104ee08c680 [CANDIDATE]: Term 1 election: Requested vote from peers fadc366fe3d44e068aee86188773c207 (127.16.121.129:42243), c12e3d60fcc24875801961d9ddb3fe63 (127.16.121.130:43775)
I20260928 22:49:23.306329 23900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d157dc401c7421cb8106ec5723d507b" candidate_uuid: "e1ab1de193eb40a9bb803104ee08c680" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fadc366fe3d44e068aee86188773c207"
I20260928 22:49:23.306425 23900 raft_consensus.cc:3037] T 6d157dc401c7421cb8106ec5723d507b P fadc366fe3d44e068aee86188773c207 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:23.306576 24037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d157dc401c7421cb8106ec5723d507b" candidate_uuid: "e1ab1de193eb40a9bb803104ee08c680" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12e3d60fcc24875801961d9ddb3fe63"
I20260928 22:49:23.306710 24037 raft_consensus.cc:3037] T 6d157dc401c7421cb8106ec5723d507b P c12e3d60fcc24875801961d9ddb3fe63 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:23.307364 23900 raft_consensus.cc:2454] T 6d157dc401c7421cb8106ec5723d507b P fadc366fe3d44e068aee86188773c207 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1ab1de193eb40a9bb803104ee08c680 in term 1.
I20260928 22:49:23.307559 24109 leader_election.cc:302] T 6d157dc401c7421cb8106ec5723d507b P e1ab1de193eb40a9bb803104ee08c680 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e1ab1de193eb40a9bb803104ee08c680, fadc366fe3d44e068aee86188773c207; no voters:
I20260928 22:49:23.307631 24037 raft_consensus.cc:2454] T 6d157dc401c7421cb8106ec5723d507b P c12e3d60fcc24875801961d9ddb3fe63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1ab1de193eb40a9bb803104ee08c680 in term 1.
I20260928 22:49:23.307672 24252 raft_consensus.cc:2781] T 6d157dc401c7421cb8106ec5723d507b P e1ab1de193eb40a9bb803104ee08c680 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:49:23.307870 24252 raft_consensus.cc:686] T 6d157dc401c7421cb8106ec5723d507b P e1ab1de193eb40a9bb803104ee08c680 [term 1 LEADER]: Becoming Leader. State: Replica: e1ab1de193eb40a9bb803104ee08c680, State: Running, Role: LEADER
I20260928 22:49:23.307996 24252 consensus_queue.cc:237] T 6d157dc401c7421cb8106ec5723d507b P e1ab1de193eb40a9bb803104ee08c680 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fadc366fe3d44e068aee86188773c207" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 42243 } } peers { permanent_uuid: "e1ab1de193eb40a9bb803104ee08c680" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36269 } } peers { permanent_uuid: "c12e3d60fcc24875801961d9ddb3fe63" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 43775 } }
I20260928 22:49:23.309059 22982 catalog_manager.cc:5495] T 6d157dc401c7421cb8106ec5723d507b P e1ab1de193eb40a9bb803104ee08c680 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1ab1de193eb40a9bb803104ee08c680 (127.16.121.131). New cstate: current_term: 1 leader_uuid: "e1ab1de193eb40a9bb803104ee08c680" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fadc366fe3d44e068aee86188773c207" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 42243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1ab1de193eb40a9bb803104ee08c680" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c12e3d60fcc24875801961d9ddb3fe63" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 43775 } health_report { overall_health: UNKNOWN } } }
I20260928 22:49:23.309687 23729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6c2ca4515ad9482a8d102db695d461ec" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } peers { permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 46125 } attrs { promote: false } } peers { permanent_uuid: "f7b7f09129f5436ab464f41cbfd8c38f" member_type: NON_VOTER last_known_addr { host: "127.16.121.188" port: 32847 } attrs { promote: true } } }
I20260928 22:49:23.309803 23729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec [sys.catalog]: This master's current role is: LEADER
I20260928 22:49:23.311136 23729 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec: attempting to promote NON_VOTER f7b7f09129f5436ab464f41cbfd8c38f to VOTER
I20260928 22:49:23.311283 23729 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } peers { permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 46125 } attrs { promote: false } } peers { permanent_uuid: "f7b7f09129f5436ab464f41cbfd8c38f" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 32847 } attrs { promote: false } }
I20260928 22:49:23.311724 23589 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [term 1 FOLLOWER]: Refusing update from remote peer 6c2ca4515ad9482a8d102db695d461ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260928 22:49:23.312062 23777 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f [term 1 LEARNER]: Refusing update from remote peer 6c2ca4515ad9482a8d102db695d461ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260928 22:49:23.312232 23731 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 46125 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260928 22:49:23.312353 23730 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7b7f09129f5436ab464f41cbfd8c38f" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 32847 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260928 22:49:23.313161 23731 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, f7b7f09129f5436ab464f41cbfd8c38f (127.16.121.188) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } peers { permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 46125 } attrs { promote: false } } peers { permanent_uuid: "f7b7f09129f5436ab464f41cbfd8c38f" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 32847 } attrs { promote: false } } }
I20260928 22:49:23.313277 23589 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, f7b7f09129f5436ab464f41cbfd8c38f (127.16.121.188) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } peers { permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 46125 } attrs { promote: false } } peers { permanent_uuid: "f7b7f09129f5436ab464f41cbfd8c38f" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 32847 } attrs { promote: false } } }
I20260928 22:49:23.313341 23777 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, f7b7f09129f5436ab464f41cbfd8c38f (127.16.121.188) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } peers { permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 46125 } attrs { promote: false } } peers { permanent_uuid: "f7b7f09129f5436ab464f41cbfd8c38f" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 32847 } attrs { promote: false } } }
I20260928 22:49:23.314162 23731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6c2ca4515ad9482a8d102db695d461ec" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } peers { permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 46125 } attrs { promote: false } } peers { permanent_uuid: "f7b7f09129f5436ab464f41cbfd8c38f" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 32847 } attrs { promote: false } } }
I20260928 22:49:23.314248 23731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec [sys.catalog]: This master's current role is: LEADER
I20260928 22:49:23.314306 23731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6c2ca4515ad9482a8d102db695d461ec" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } peers { permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 46125 } attrs { promote: false } } peers { permanent_uuid: "f7b7f09129f5436ab464f41cbfd8c38f" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 32847 } attrs { promote: false } } }
I20260928 22:49:23.314359 23731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec [sys.catalog]: This master's current role is: LEADER
I20260928 22:49:23.314407 23731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6c2ca4515ad9482a8d102db695d461ec" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } peers { permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 46125 } attrs { promote: false } } peers { permanent_uuid: "f7b7f09129f5436ab464f41cbfd8c38f" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 32847 } attrs { promote: false } } }
I20260928 22:49:23.314455 23731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec [sys.catalog]: This master's current role is: LEADER
I20260928 22:49:23.314486 23806 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c2ca4515ad9482a8d102db695d461ec" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } peers { permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 46125 } attrs { promote: false } } peers { permanent_uuid: "f7b7f09129f5436ab464f41cbfd8c38f" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 32847 } attrs { promote: false } } }
I20260928 22:49:23.314563 23806 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:49:23.314661 24261 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c2ca4515ad9482a8d102db695d461ec" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } peers { permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 46125 } attrs { promote: false } } peers { permanent_uuid: "f7b7f09129f5436ab464f41cbfd8c38f" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 32847 } attrs { promote: false } } }
I20260928 22:49:23.314819 24261 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:49:23.345229 24242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:23.342055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40914 (local address 127.16.121.131:36269)
0928 22:49:23.342156 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:23.342157 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:23.342166 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:49:23.342203 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:23.342204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:23.342231 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:23.342317 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:49:23.342663 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.343374 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.344240 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.344496 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.344573 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:23.344845 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:49:23.344961 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:49:23.345044 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:23.345139 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 22:49:23.345162 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 22:49:23.347529 24037 raft_consensus.cc:1261] T 6d157dc401c7421cb8106ec5723d507b P c12e3d60fcc24875801961d9ddb3fe63 [term 1 FOLLOWER]: Refusing update from remote peer e1ab1de193eb40a9bb803104ee08c680: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:49:23.347576 23900 raft_consensus.cc:1261] T 6d157dc401c7421cb8106ec5723d507b P fadc366fe3d44e068aee86188773c207 [term 1 FOLLOWER]: Refusing update from remote peer e1ab1de193eb40a9bb803104ee08c680: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:49:23.347945 24258 consensus_queue.cc:1035] T 6d157dc401c7421cb8106ec5723d507b P e1ab1de193eb40a9bb803104ee08c680 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fadc366fe3d44e068aee86188773c207" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 42243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:49:23.348088 24252 consensus_queue.cc:1035] T 6d157dc401c7421cb8106ec5723d507b P e1ab1de193eb40a9bb803104ee08c680 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12e3d60fcc24875801961d9ddb3fe63" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 43775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 22:49:23.350425 23949 tablet.cc:2369] T 6d157dc401c7421cb8106ec5723d507b P fadc366fe3d44e068aee86188773c207: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:49:23.351311 24263 mvcc.cc:205] Tried to move back new op lower bound from 7334444086668075008 to 7334444086510202880. Current Snapshot: MvccSnapshot[applied={T|T < 7334444086668075008}]
I20260928 22:49:23.352450 24265 mvcc.cc:205] Tried to move back new op lower bound from 7334444086668075008 to 7334444086510202880. Current Snapshot: MvccSnapshot[applied={T|T < 7334444086668075008 or (T in {7334444086668075008})}]
I20260928 22:49:23.355214 22982 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:49:23.355366 22982 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:49:23.355777 24264 mvcc.cc:205] Tried to move back new op lower bound from 7334444086668075008 to 7334444086510202880. Current Snapshot: MvccSnapshot[applied={T|T < 7334444086668075008 or (T in {7334444086668075008})}]
I20260928 22:49:23.357293 22982 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c2ca4515ad9482a8d102db695d461ec: Sending DeleteTablet for 3 replicas of tablet 6d157dc401c7421cb8106ec5723d507b
I20260928 22:49:23.357771 24153 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d157dc401c7421cb8106ec5723d507b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:49:23 UTC) from {username='oryx', principal='oryx/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:40900
I20260928 22:49:23.358007 23880 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d157dc401c7421cb8106ec5723d507b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:49:23 UTC) from {username='oryx', principal='oryx/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:41398
I20260928 22:49:23.357996 24017 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d157dc401c7421cb8106ec5723d507b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:49:23 UTC) from {username='oryx', principal='oryx/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:42820
I20260928 22:49:23.358180 24273 tablet_replica.cc:330] T 6d157dc401c7421cb8106ec5723d507b P e1ab1de193eb40a9bb803104ee08c680: stopping tablet replica
I20260928 22:49:23.358258 24274 tablet_replica.cc:330] T 6d157dc401c7421cb8106ec5723d507b P fadc366fe3d44e068aee86188773c207: stopping tablet replica
I20260928 22:49:23.358371 24274 raft_consensus.cc:2229] T 6d157dc401c7421cb8106ec5723d507b P fadc366fe3d44e068aee86188773c207 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:49:23.358301 24273 raft_consensus.cc:2229] T 6d157dc401c7421cb8106ec5723d507b P e1ab1de193eb40a9bb803104ee08c680 [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:49:23.358486 24274 raft_consensus.cc:2258] T 6d157dc401c7421cb8106ec5723d507b P fadc366fe3d44e068aee86188773c207 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:49:23.358633 24273 raft_consensus.cc:2258] T 6d157dc401c7421cb8106ec5723d507b P e1ab1de193eb40a9bb803104ee08c680 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:49:23.359081 24274 ts_tablet_manager.cc:1905] T 6d157dc401c7421cb8106ec5723d507b P fadc366fe3d44e068aee86188773c207: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:49:23.359161 24273 ts_tablet_manager.cc:1905] T 6d157dc401c7421cb8106ec5723d507b P e1ab1de193eb40a9bb803104ee08c680: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:49:23.359356 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 22950
I20260928 22:49:23.359690 24275 tablet_replica.cc:330] T 6d157dc401c7421cb8106ec5723d507b P c12e3d60fcc24875801961d9ddb3fe63: stopping tablet replica
I20260928 22:49:23.359853 24275 raft_consensus.cc:2229] T 6d157dc401c7421cb8106ec5723d507b P c12e3d60fcc24875801961d9ddb3fe63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:49:23.360077 24275 raft_consensus.cc:2258] T 6d157dc401c7421cb8106ec5723d507b P c12e3d60fcc24875801961d9ddb3fe63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:49:23.360610 24275 ts_tablet_manager.cc:1905] T 6d157dc401c7421cb8106ec5723d507b P c12e3d60fcc24875801961d9ddb3fe63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:49:23.363106 24275 ts_tablet_manager.cc:1918] T 6d157dc401c7421cb8106ec5723d507b P c12e3d60fcc24875801961d9ddb3fe63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:49:23.363291 24275 log.cc:1196] T 6d157dc401c7421cb8106ec5723d507b P c12e3d60fcc24875801961d9ddb3fe63: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-1/wal/wals/6d157dc401c7421cb8106ec5723d507b
I20260928 22:49:23.363731 24275 ts_tablet_manager.cc:1939] T 6d157dc401c7421cb8106ec5723d507b P c12e3d60fcc24875801961d9ddb3fe63: Deleting consensus metadata
I20260928 22:49:23.364310 24274 ts_tablet_manager.cc:1918] T 6d157dc401c7421cb8106ec5723d507b P fadc366fe3d44e068aee86188773c207: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:49:23.364425 24274 log.cc:1196] T 6d157dc401c7421cb8106ec5723d507b P fadc366fe3d44e068aee86188773c207: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-0/wal/wals/6d157dc401c7421cb8106ec5723d507b
I20260928 22:49:23.364724 24274 ts_tablet_manager.cc:1939] T 6d157dc401c7421cb8106ec5723d507b P fadc366fe3d44e068aee86188773c207: Deleting consensus metadata
I20260928 22:49:23.364989 24273 ts_tablet_manager.cc:1918] T 6d157dc401c7421cb8106ec5723d507b P e1ab1de193eb40a9bb803104ee08c680: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:49:23.365083 24273 log.cc:1196] T 6d157dc401c7421cb8106ec5723d507b P e1ab1de193eb40a9bb803104ee08c680: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-2/wal/wals/6d157dc401c7421cb8106ec5723d507b
I20260928 22:49:23.367014 24273 ts_tablet_manager.cc:1939] T 6d157dc401c7421cb8106ec5723d507b P e1ab1de193eb40a9bb803104ee08c680: Deleting consensus metadata
W20260928 22:49:23.368022 24106 connection.cc:535] server connection from 127.0.0.1:40900 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 22:49:23.368261 23973 connection.cc:535] server connection from 127.0.0.1:42820 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 22:49:23.368381 23833 connection.cc:535] server connection from 127.0.0.1:41398 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 22:49:23.372187 24234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:23.359156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33396 (local address 127.16.121.188:32847)
0928 22:49:23.359208 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:23.359211 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:23.359220 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:49:23.359245 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:23.359246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:23.359258 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:23.359311 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:23.362598 (+ 3287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.363478 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.365376 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.365780 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.365832 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:23.368525 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:23.368527 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:23.368527 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:23.368537 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:23.370540 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:23.371129 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:23.371130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:23.371131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:23.371198 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:23.371449 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:23.371449 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:23.371450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:23.371609 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:23.371717 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:23.371983 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 22:49:23.372079 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 22:49:23.380271 23952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:23.370337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33398 (local address 127.16.121.188:32847)
0928 22:49:23.370382 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:23.370384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:23.370754 (+ 370us) server_negotiation.cc:407] Connection header received
0928 22:49:23.370851 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:23.370852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:23.370863 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:23.370894 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:23.371857 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.372561 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.373756 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.374119 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.374202 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:23.376310 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:23.376312 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:23.376312 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:23.376321 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:23.378680 (+ 2359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:23.379265 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:23.379266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:23.379266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:23.379427 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:23.379650 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:23.379651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:23.379651 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:23.379804 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:23.379905 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:23.380078 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 22:49:23.380170 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:49:23.380344 24232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:23.370437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49922 (local address 127.16.121.189:46125)
0928 22:49:23.370479 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:23.370481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:23.370586 (+ 105us) server_negotiation.cc:407] Connection header received
0928 22:49:23.370670 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:23.370672 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:23.370688 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:23.370743 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:23.371578 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.372490 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.373780 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:23.374017 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:23.374053 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:23.376323 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:23.376325 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:23.376325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:23.376334 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:23.378513 (+ 2179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:23.379250 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:23.379252 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:23.379252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:23.379320 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:23.379558 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:23.379559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:23.379559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:23.379725 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:23.379802 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:23.380113 (+ 311us) server_negotiation.cc:299] Negotiation successful
0928 22:49:23.380220 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:49:24.355685 24285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:24.355436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:39331 (local address 127.16.121.131:54967)
0928 22:49:24.355557 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:24.355633 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:39331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":57,"threads_started":1}
I20260928 22:49:24.355676 24284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:24.355361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:39331 (local address 127.16.121.130:54341)
0928 22:49:24.355530 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:24.355615 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:39331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":94,"threads_started":1}
W20260928 22:49:24.355890 24084 heartbeater.cc:644] Failed to heartbeat to 127.16.121.190:39331 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.121.190:39331: connect: Connection refused (error 111)
W20260928 22:49:24.355908 24220 heartbeater.cc:644] Failed to heartbeat to 127.16.121.190:39331 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.121.190:39331: connect: Connection refused (error 111)
I20260928 22:49:24.356170 24286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:24.355597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:39331 (local address 127.16.121.129:45683)
0928 22:49:24.356049 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:24.356125 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:39331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":50,"threads_started":1}
W20260928 22:49:24.356386 23947 heartbeater.cc:644] Failed to heartbeat to 127.16.121.190:39331 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.121.190:39331: connect: Connection refused (error 111)
I20260928 22:49:24.857334 24287 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6c2ca4515ad9482a8d102db695d461ec)
I20260928 22:49:24.857477 24287 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } peers { permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 46125 } attrs { promote: false } } peers { permanent_uuid: "f7b7f09129f5436ab464f41cbfd8c38f" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 32847 } attrs { promote: false } }
I20260928 22:49:24.857751 24287 leader_election.cc:288] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6c2ca4515ad9482a8d102db695d461ec (127.16.121.190:39331), f7b7f09129f5436ab464f41cbfd8c38f (127.16.121.188:32847)
I20260928 22:49:24.858158 24288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:24.857876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:39331 (local address 127.0.0.1:47408)
0928 22:49:24.858046 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:24.858114 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:39331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":88,"threads_started":1}
I20260928 22:49:24.858196 24289 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6c2ca4515ad9482a8d102db695d461ec)
I20260928 22:49:24.858299 24289 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } peers { permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 46125 } attrs { promote: false } } peers { permanent_uuid: "f7b7f09129f5436ab464f41cbfd8c38f" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 32847 } attrs { promote: false } }
W20260928 22:49:24.858461 23565 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.121.190:39331: connect: Connection refused (error 111)
I20260928 22:49:24.858920 24293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:24.858704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:39331 (local address 127.0.0.1:47422)
0928 22:49:24.858812 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:24.858872 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:39331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":64,"threads_started":1}
I20260928 22:49:24.858951 24288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:24.858857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:39331 (local address 127.0.0.1:47426)
0928 22:49:24.858898 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:24.858923 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:39331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 22:49:24.859014 23565 leader_election.cc:334] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6c2ca4515ad9482a8d102db695d461ec (127.16.121.190:39331): Network error: Client connection negotiation failed: client connection to 127.16.121.190:39331: connect: Connection refused (error 111)
W20260928 22:49:24.859401 23753 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.121.190:39331: connect: Connection refused (error 111)
I20260928 22:49:24.859623 24289 leader_election.cc:288] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6c2ca4515ad9482a8d102db695d461ec (127.16.121.190:39331), a5fee57a42a14e1ab90bc3db2e453559 (127.16.121.189:46125)
I20260928 22:49:24.859929 24294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:24.859812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:39331 (local address 127.0.0.1:47442)
0928 22:49:24.859849 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:24.859892 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:39331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260928 22:49:24.860106 23753 leader_election.cc:334] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6c2ca4515ad9482a8d102db695d461ec (127.16.121.190:39331): Network error: Client connection negotiation failed: client connection to 127.16.121.190:39331: connect: Connection refused (error 111)
I20260928 22:49:24.861789 24291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:24.858129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33412 (local address 127.16.121.188:32847)
0928 22:49:24.858253 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:24.858255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:24.858265 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:49:24.858295 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:24.858296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:24.858307 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:24.858388 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:49:24.858605 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:24.859500 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:24.861059 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:24.861452 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:24.861542 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:24.861552 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:24.861681 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 22:49:24.861720 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":48,"threads_started":1}
I20260928 22:49:24.861789 24290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:24.857968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:32847 (local address 127.0.0.1:33412)
0928 22:49:24.858096 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:24.858108 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:24.858177 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:24.858396 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:24.858397 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:24.858401 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:49:24.858476 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:24.858479 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:24.859609 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:24.859610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:24.860978 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:24.860980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:24.861541 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:24.861542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:24.861605 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:24.861609 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:49:24.861665 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 22:49:24.861716 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":68,"threads_started":1}
I20260928 22:49:24.862290 23777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a5fee57a42a14e1ab90bc3db2e453559" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f7b7f09129f5436ab464f41cbfd8c38f" is_pre_election: true
I20260928 22:49:24.862429 23777 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5fee57a42a14e1ab90bc3db2e453559 in term 1.
I20260928 22:49:24.862701 24295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:24.859200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49938 (local address 127.16.121.189:46125)
0928 22:49:24.859380 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:24.859382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:24.859391 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:49:24.859517 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:24.859518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:24.859533 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:24.859786 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:49:24.859958 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:24.860702 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:24.861828 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:24.862350 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:24.862439 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:24.862446 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:24.862598 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 22:49:24.862636 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":121,"server-negotiator.queue_time_us":145,"thread_start_us":102,"threads_started":1}
I20260928 22:49:24.862702 24293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:24.859036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.189:46125 (local address 127.0.0.1:49938)
0928 22:49:24.859078 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:24.859089 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:24.859144 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:24.859659 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:24.859660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:24.859664 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:49:24.859715 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:24.859717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:24.860827 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:24.860829 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:24.861706 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:24.861709 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:24.862441 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:24.862442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:24.862510 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:24.862514 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:49:24.862591 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 22:49:24.862636 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"thread_start_us":79,"threads_started":1}
I20260928 22:49:24.862864 23563 leader_election.cc:302] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a5fee57a42a14e1ab90bc3db2e453559, f7b7f09129f5436ab464f41cbfd8c38f; no voters: 6c2ca4515ad9482a8d102db695d461ec
I20260928 22:49:24.862970 24287 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 22:49:24.863021 24287 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6c2ca4515ad9482a8d102db695d461ec)
I20260928 22:49:24.863065 24287 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [term 1 FOLLOWER]: Advancing to term 2
I20260928 22:49:24.863070 23589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f7b7f09129f5436ab464f41cbfd8c38f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a5fee57a42a14e1ab90bc3db2e453559" is_pre_election: true
I20260928 22:49:24.863871 24287 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } peers { permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 46125 } attrs { promote: false } } peers { permanent_uuid: "f7b7f09129f5436ab464f41cbfd8c38f" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 32847 } attrs { promote: false } }
I20260928 22:49:24.864027 23589 raft_consensus.cc:2379] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7b7f09129f5436ab464f41cbfd8c38f in current term 2: Already voted for candidate a5fee57a42a14e1ab90bc3db2e453559 in this term.
I20260928 22:49:24.864056 24287 leader_election.cc:288] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [CANDIDATE]: Term 2 election: Requested vote from peers 6c2ca4515ad9482a8d102db695d461ec (127.16.121.190:39331), f7b7f09129f5436ab464f41cbfd8c38f (127.16.121.188:32847)
I20260928 22:49:24.864205 23777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a5fee57a42a14e1ab90bc3db2e453559" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f7b7f09129f5436ab464f41cbfd8c38f"
I20260928 22:49:24.864240 24290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:24.864141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:39331 (local address 127.0.0.1:47446)
0928 22:49:24.864177 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:24.864207 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:39331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260928 22:49:24.864311 23777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f [term 1 FOLLOWER]: Advancing to term 2
I20260928 22:49:24.864254 23751 leader_election.cc:302] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7b7f09129f5436ab464f41cbfd8c38f; no voters: 6c2ca4515ad9482a8d102db695d461ec, a5fee57a42a14e1ab90bc3db2e453559
I20260928 22:49:24.864565 24290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:24.864481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:39331 (local address 127.0.0.1:47462)
0928 22:49:24.864522 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:24.864548 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:39331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260928 22:49:24.864651 23565 leader_election.cc:334] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6c2ca4515ad9482a8d102db695d461ec (127.16.121.190:39331): Network error: Client connection negotiation failed: client connection to 127.16.121.190:39331: connect: Connection refused (error 111)
I20260928 22:49:24.865135 23777 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5fee57a42a14e1ab90bc3db2e453559 in term 2.
I20260928 22:49:24.865221 24289 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 22:49:24.865345 23563 leader_election.cc:302] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a5fee57a42a14e1ab90bc3db2e453559, f7b7f09129f5436ab464f41cbfd8c38f; no voters: 6c2ca4515ad9482a8d102db695d461ec
I20260928 22:49:24.865440 24287 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 22:49:24.865603 24287 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [term 2 LEADER]: Becoming Leader. State: Replica: a5fee57a42a14e1ab90bc3db2e453559, State: Running, Role: LEADER
I20260928 22:49:24.865667 24287 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } peers { permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 46125 } attrs { promote: false } } peers { permanent_uuid: "f7b7f09129f5436ab464f41cbfd8c38f" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 32847 } attrs { promote: false } }
I20260928 22:49:24.866045 24297 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5fee57a42a14e1ab90bc3db2e453559. Latest consensus state: current_term: 2 leader_uuid: "a5fee57a42a14e1ab90bc3db2e453559" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } peers { permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 46125 } attrs { promote: false } } peers { permanent_uuid: "f7b7f09129f5436ab464f41cbfd8c38f" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 32847 } attrs { promote: false } } }
I20260928 22:49:24.866123 24297 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [sys.catalog]: This master's current role is: LEADER
I20260928 22:49:24.866299 24299 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:49:24.866818 24299 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:49:24.866909 24299 catalog_manager.cc:1259] Loaded cluster ID: f581d50fabd146eab37464bd687b1b1c
I20260928 22:49:24.866926 24299 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:49:24.867219 24299 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:49:24.867300 24299 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559: Loaded TSK: 0
I20260928 22:49:24.867465 24299 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:49:24.882258 23579 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49922:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:49:24.884138 23579 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:49:24.885525 23777 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f [term 2 FOLLOWER]: Refusing update from remote peer a5fee57a42a14e1ab90bc3db2e453559: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260928 22:49:24.885703 24290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:24.885437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:39331 (local address 127.0.0.1:47484)
0928 22:49:24.885573 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:24.885623 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:39331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 22:49:24.885859 24287 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7b7f09129f5436ab464f41cbfd8c38f" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 32847 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260928 22:49:24.886317 24290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:24.886114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:39331 (local address 127.0.0.1:47500)
0928 22:49:24.886243 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:24.886291 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:39331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 22:49:24.886462 23565 consensus_peers.cc:487] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 -> Peer 6c2ca4515ad9482a8d102db695d461ec (127.16.121.190:39331): Couldn't send request to peer 6c2ca4515ad9482a8d102db695d461ec. Status: Network error: Client connection negotiation failed: client connection to 127.16.121.190:39331: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 22:49:24.887014 24289 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5fee57a42a14e1ab90bc3db2e453559. Latest consensus state: current_term: 2 leader_uuid: "a5fee57a42a14e1ab90bc3db2e453559" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } peers { permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 46125 } attrs { promote: false } } peers { permanent_uuid: "f7b7f09129f5436ab464f41cbfd8c38f" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 32847 } attrs { promote: false } } }
I20260928 22:49:24.887187 24289 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:49:24.887845 24297 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "a5fee57a42a14e1ab90bc3db2e453559" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } peers { permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 46125 } attrs { promote: false } } peers { permanent_uuid: "f7b7f09129f5436ab464f41cbfd8c38f" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 32847 } attrs { promote: false } } }
I20260928 22:49:24.887967 24297 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559 [sys.catalog]: This master's current role is: LEADER
I20260928 22:49:24.888011 24289 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "a5fee57a42a14e1ab90bc3db2e453559" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6c2ca4515ad9482a8d102db695d461ec" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39331 } } peers { permanent_uuid: "a5fee57a42a14e1ab90bc3db2e453559" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 46125 } attrs { promote: false } } peers { permanent_uuid: "f7b7f09129f5436ab464f41cbfd8c38f" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 32847 } attrs { promote: false } } }
I20260928 22:49:24.888113 24289 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7b7f09129f5436ab464f41cbfd8c38f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:49:24.888325 24301 mvcc.cc:205] Tried to move back new op lower bound from 7334444092968132608 to 7334444092890488832. Current Snapshot: MvccSnapshot[applied={T|T < 7334444092968132608}]
I20260928 22:49:24.893565 24307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:24.890097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40924 (local address 127.16.121.131:36269)
0928 22:49:24.890392 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:24.890394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:24.890402 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:49:24.890430 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:24.890430 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:24.890441 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:24.890647 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:49:24.890690 (+ 43us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:24.891276 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:24.892660 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:24.893242 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:24.893307 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:24.893314 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:24.893465 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 22:49:24.893505 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":53,"threads_started":1}
I20260928 22:49:24.893577 24290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:24.889809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.131:36269 (local address 127.0.0.1:40924)
0928 22:49:24.889893 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:24.889957 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:24.890000 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:24.890531 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:24.890531 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:24.890533 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:49:24.890581 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:24.890583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:24.891357 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:24.891358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:24.892548 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:24.892551 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:24.893316 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:24.893316 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:24.893381 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:24.893386 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:49:24.893452 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 22:49:24.893495 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 22:49:24.893872 24153 tablet_service.cc:1467] Processing CreateTablet for tablet 7ddbef8a3bcc4a0e8726a97340dd555b (DEFAULT_TABLE table=test-table [id=0693468d6d064a47a2d84ffb0eb16b62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:49:24.894025 24153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ddbef8a3bcc4a0e8726a97340dd555b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:24.894811 24288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:24.889899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:42243 (local address 127.0.0.1:41414)
0928 22:49:24.889935 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:24.889946 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:24.890015 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:24.890524 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:24.890525 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:24.890527 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:49:24.890556 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:24.890557 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:24.891521 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:24.891521 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:24.892570 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:24.892572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:24.894492 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:24.894494 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:24.894559 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:24.894562 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:49:24.894683 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 22:49:24.894735 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20,"mutex_wait_us":1}
I20260928 22:49:24.894829 24308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:24.890241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41414 (local address 127.16.121.129:42243)
0928 22:49:24.890381 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:24.890383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:24.890391 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:49:24.890417 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:24.890418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:24.890428 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:24.890620 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:49:24.890660 (+ 40us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:24.891439 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:24.893812 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:24.894388 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:24.894473 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:24.894482 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:24.894698 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 22:49:24.894758 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":57,"threads_started":1}
I20260928 22:49:24.895300 23880 tablet_service.cc:1467] Processing CreateTablet for tablet 7ddbef8a3bcc4a0e8726a97340dd555b (DEFAULT_TABLE table=test-table [id=0693468d6d064a47a2d84ffb0eb16b62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:49:24.895440 23880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ddbef8a3bcc4a0e8726a97340dd555b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:24.895581 24305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:24.890013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.130:43775 (local address 127.0.0.1:42836)
0928 22:49:24.890209 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:24.890218 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:24.890284 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:24.890487 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:24.890488 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:24.890491 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:49:24.890549 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:24.890551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:24.892654 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:24.892654 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:24.893679 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:24.893682 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:24.895267 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:24.895268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:24.895338 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:24.895343 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:49:24.895393 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 22:49:24.895451 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":125,"threads_started":1}
I20260928 22:49:24.895864 24306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:24.890091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42836 (local address 127.16.121.130:43775)
0928 22:49:24.890190 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:24.890192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:24.890270 (+ 78us) server_negotiation.cc:407] Connection header received
0928 22:49:24.890349 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:24.890350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:24.890362 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:24.890429 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:49:24.890630 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:24.891439 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:24.894528 (+ 3089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:24.895147 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:24.895223 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:24.895231 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:24.895731 (+ 500us) server_negotiation.cc:299] Negotiation successful
0928 22:49:24.895792 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":57,"threads_started":1}
I20260928 22:49:24.896030 24309 tablet_bootstrap.cc:492] T 7ddbef8a3bcc4a0e8726a97340dd555b P e1ab1de193eb40a9bb803104ee08c680: Bootstrap starting.
I20260928 22:49:24.896128 24017 tablet_service.cc:1467] Processing CreateTablet for tablet 7ddbef8a3bcc4a0e8726a97340dd555b (DEFAULT_TABLE table=test-table [id=0693468d6d064a47a2d84ffb0eb16b62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:49:24.896246 24017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ddbef8a3bcc4a0e8726a97340dd555b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:24.896384 24309 tablet_bootstrap.cc:654] T 7ddbef8a3bcc4a0e8726a97340dd555b P e1ab1de193eb40a9bb803104ee08c680: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:24.896948 24309 tablet_bootstrap.cc:492] T 7ddbef8a3bcc4a0e8726a97340dd555b P e1ab1de193eb40a9bb803104ee08c680: No bootstrap required, opened a new log
I20260928 22:49:24.896958 24311 tablet_bootstrap.cc:492] T 7ddbef8a3bcc4a0e8726a97340dd555b P fadc366fe3d44e068aee86188773c207: Bootstrap starting.
I20260928 22:49:24.896998 24309 ts_tablet_manager.cc:1397] T 7ddbef8a3bcc4a0e8726a97340dd555b P e1ab1de193eb40a9bb803104ee08c680: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:24.897119 24309 raft_consensus.cc:348] T 7ddbef8a3bcc4a0e8726a97340dd555b P e1ab1de193eb40a9bb803104ee08c680 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1ab1de193eb40a9bb803104ee08c680" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36269 } } peers { permanent_uuid: "fadc366fe3d44e068aee86188773c207" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 42243 } } peers { permanent_uuid: "c12e3d60fcc24875801961d9ddb3fe63" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 43775 } }
I20260928 22:49:24.897228 24309 raft_consensus.cc:374] T 7ddbef8a3bcc4a0e8726a97340dd555b P e1ab1de193eb40a9bb803104ee08c680 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:24.897274 24309 raft_consensus.cc:729] T 7ddbef8a3bcc4a0e8726a97340dd555b P e1ab1de193eb40a9bb803104ee08c680 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1ab1de193eb40a9bb803104ee08c680, State: Initialized, Role: FOLLOWER
I20260928 22:49:24.897316 24311 tablet_bootstrap.cc:654] T 7ddbef8a3bcc4a0e8726a97340dd555b P fadc366fe3d44e068aee86188773c207: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:24.897365 24309 consensus_queue.cc:260] T 7ddbef8a3bcc4a0e8726a97340dd555b P e1ab1de193eb40a9bb803104ee08c680 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1ab1de193eb40a9bb803104ee08c680" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36269 } } peers { permanent_uuid: "fadc366fe3d44e068aee86188773c207" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 42243 } } peers { permanent_uuid: "c12e3d60fcc24875801961d9ddb3fe63" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 43775 } }
I20260928 22:49:24.897574 24309 ts_tablet_manager.cc:1428] T 7ddbef8a3bcc4a0e8726a97340dd555b P e1ab1de193eb40a9bb803104ee08c680: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:24.898144 24315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:24.897871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:39331 (local address 127.16.121.131:46653)
0928 22:49:24.898067 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:24.898106 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:39331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":108,"threads_started":1}
I20260928 22:49:24.898250 24221 heartbeater.cc:503] Master 127.16.121.189:46125 was elected leader, sending a full tablet report...
I20260928 22:49:24.898413 24311 tablet_bootstrap.cc:492] T 7ddbef8a3bcc4a0e8726a97340dd555b P fadc366fe3d44e068aee86188773c207: No bootstrap required, opened a new log
I20260928 22:49:24.898476 24311 ts_tablet_manager.cc:1397] T 7ddbef8a3bcc4a0e8726a97340dd555b P fadc366fe3d44e068aee86188773c207: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:49:24.898671 24311 raft_consensus.cc:348] T 7ddbef8a3bcc4a0e8726a97340dd555b P fadc366fe3d44e068aee86188773c207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1ab1de193eb40a9bb803104ee08c680" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36269 } } peers { permanent_uuid: "fadc366fe3d44e068aee86188773c207" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 42243 } } peers { permanent_uuid: "c12e3d60fcc24875801961d9ddb3fe63" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 43775 } }
I20260928 22:49:24.898770 24315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:24.898485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:39331 (local address 127.16.121.131:47635)
0928 22:49:24.898707 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:24.898735 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:39331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":90,"threads_started":1}
I20260928 22:49:24.898834 24311 raft_consensus.cc:374] T 7ddbef8a3bcc4a0e8726a97340dd555b P fadc366fe3d44e068aee86188773c207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:24.898909 24311 raft_consensus.cc:729] T 7ddbef8a3bcc4a0e8726a97340dd555b P fadc366fe3d44e068aee86188773c207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fadc366fe3d44e068aee86188773c207, State: Initialized, Role: FOLLOWER
I20260928 22:49:24.898976 24311 consensus_queue.cc:260] T 7ddbef8a3bcc4a0e8726a97340dd555b P fadc366fe3d44e068aee86188773c207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1ab1de193eb40a9bb803104ee08c680" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36269 } } peers { permanent_uuid: "fadc366fe3d44e068aee86188773c207" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 42243 } } peers { permanent_uuid: "c12e3d60fcc24875801961d9ddb3fe63" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 43775 } }
I20260928 22:49:24.899026 24314 tablet_bootstrap.cc:492] T 7ddbef8a3bcc4a0e8726a97340dd555b P c12e3d60fcc24875801961d9ddb3fe63: Bootstrap starting.
I20260928 22:49:24.899180 24311 ts_tablet_manager.cc:1428] T 7ddbef8a3bcc4a0e8726a97340dd555b P fadc366fe3d44e068aee86188773c207: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 22:49:24.899551 24314 tablet_bootstrap.cc:654] T 7ddbef8a3bcc4a0e8726a97340dd555b P c12e3d60fcc24875801961d9ddb3fe63: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:24.899698 24318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:24.899465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:39331 (local address 127.16.121.129:47107)
0928 22:49:24.899615 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:24.899650 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:39331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":90,"threads_started":1}
I20260928 22:49:24.900061 23948 heartbeater.cc:503] Master 127.16.121.189:46125 was elected leader, sending a full tablet report...
I20260928 22:49:24.900426 24314 tablet_bootstrap.cc:492] T 7ddbef8a3bcc4a0e8726a97340dd555b P c12e3d60fcc24875801961d9ddb3fe63: No bootstrap required, opened a new log
I20260928 22:49:24.900473 24318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:24.900147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:39331 (local address 127.16.121.129:47183)
0928 22:49:24.900413 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:24.900445 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:39331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245}
I20260928 22:49:24.900480 24314 ts_tablet_manager.cc:1397] T 7ddbef8a3bcc4a0e8726a97340dd555b P c12e3d60fcc24875801961d9ddb3fe63: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:24.900600 24314 raft_consensus.cc:348] T 7ddbef8a3bcc4a0e8726a97340dd555b P c12e3d60fcc24875801961d9ddb3fe63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1ab1de193eb40a9bb803104ee08c680" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36269 } } peers { permanent_uuid: "fadc366fe3d44e068aee86188773c207" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 42243 } } peers { permanent_uuid: "c12e3d60fcc24875801961d9ddb3fe63" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 43775 } }
I20260928 22:49:24.900663 24314 raft_consensus.cc:374] T 7ddbef8a3bcc4a0e8726a97340dd555b P c12e3d60fcc24875801961d9ddb3fe63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:24.900677 24314 raft_consensus.cc:729] T 7ddbef8a3bcc4a0e8726a97340dd555b P c12e3d60fcc24875801961d9ddb3fe63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c12e3d60fcc24875801961d9ddb3fe63, State: Initialized, Role: FOLLOWER
I20260928 22:49:24.900713 24314 consensus_queue.cc:260] T 7ddbef8a3bcc4a0e8726a97340dd555b P c12e3d60fcc24875801961d9ddb3fe63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1ab1de193eb40a9bb803104ee08c680" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36269 } } peers { permanent_uuid: "fadc366fe3d44e068aee86188773c207" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 42243 } } peers { permanent_uuid: "c12e3d60fcc24875801961d9ddb3fe63" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 43775 } }
I20260928 22:49:24.900861 24314 ts_tablet_manager.cc:1428] T 7ddbef8a3bcc4a0e8726a97340dd555b P c12e3d60fcc24875801961d9ddb3fe63: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:24.901209 24321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:24.900984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:39331 (local address 127.16.121.130:56747)
0928 22:49:24.901114 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:24.901169 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:39331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":94,"threads_started":1}
I20260928 22:49:24.901535 24321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:24.901434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:39331 (local address 127.16.121.130:53567)
0928 22:49:24.901473 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:24.901505 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:39331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260928 22:49:24.901556 24085 heartbeater.cc:503] Master 127.16.121.189:46125 was elected leader, sending a full tablet report...
W20260928 22:49:24.910653 24222 tablet.cc:2369] T 7ddbef8a3bcc4a0e8726a97340dd555b P e1ab1de193eb40a9bb803104ee08c680: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:49:25.009182 24317 raft_consensus.cc:482] T 7ddbef8a3bcc4a0e8726a97340dd555b P fadc366fe3d44e068aee86188773c207 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:49:25.009307 24317 raft_consensus.cc:504] T 7ddbef8a3bcc4a0e8726a97340dd555b P fadc366fe3d44e068aee86188773c207 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1ab1de193eb40a9bb803104ee08c680" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36269 } } peers { permanent_uuid: "fadc366fe3d44e068aee86188773c207" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 42243 } } peers { permanent_uuid: "c12e3d60fcc24875801961d9ddb3fe63" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 43775 } }
I20260928 22:49:25.009653 24317 leader_election.cc:288] T 7ddbef8a3bcc4a0e8726a97340dd555b P fadc366fe3d44e068aee86188773c207 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1ab1de193eb40a9bb803104ee08c680 (127.16.121.131:36269), c12e3d60fcc24875801961d9ddb3fe63 (127.16.121.130:43775)
I20260928 22:49:25.013013 24318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:25.009831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.130:43775 (local address 127.16.121.129:60447)
0928 22:49:25.009895 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:25.009910 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:25.009980 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:25.010153 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:25.010154 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:25.010157 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:49:25.010236 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:25.010238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:25.011263 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:25.011264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:25.012156 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:25.012159 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:25.012735 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:25.012736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:25.012813 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:25.012816 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:49:25.012869 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 22:49:25.012913 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":1}
I20260928 22:49:25.013025 24306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:25.009971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:60447 (local address 127.16.121.130:43775)
0928 22:49:25.010019 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:25.010021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:25.010033 (+ 12us) server_negotiation.cc:407] Connection header received
0928 22:49:25.010061 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:25.010062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:25.010071 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:25.010135 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:49:25.010340 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:25.011163 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:25.012255 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:25.012647 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:25.012735 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:25.012747 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:25.012881 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 22:49:25.012936 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 22:49:25.013110 24307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:25.009932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:45231 (local address 127.16.121.131:36269)
0928 22:49:25.009975 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:25.009977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:25.010199 (+ 222us) server_negotiation.cc:407] Connection header received
0928 22:49:25.010235 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:25.010236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:25.010244 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:25.010285 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:49:25.010413 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:25.011163 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:25.012256 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:25.012645 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:25.012768 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:25.012779 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:25.012994 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 22:49:25.013037 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 22:49:25.013124 24322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:25.009831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.131:36269 (local address 127.16.121.129:45231)
0928 22:49:25.010096 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:25.010122 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:25.010176 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:25.010305 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:25.010306 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:25.010308 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:49:25.010357 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:25.010359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:25.011283 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:25.011285 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:25.012157 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:25.012159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:25.012810 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:25.012811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:25.012917 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:25.012920 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:49:25.012976 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 22:49:25.013007 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":122,"thread_start_us":122,"threads_started":1}
I20260928 22:49:25.013485 24037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ddbef8a3bcc4a0e8726a97340dd555b" candidate_uuid: "fadc366fe3d44e068aee86188773c207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12e3d60fcc24875801961d9ddb3fe63" is_pre_election: true
I20260928 22:49:25.013587 24037 raft_consensus.cc:2454] T 7ddbef8a3bcc4a0e8726a97340dd555b P c12e3d60fcc24875801961d9ddb3fe63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fadc366fe3d44e068aee86188773c207 in term 0.
I20260928 22:49:25.013564 24173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ddbef8a3bcc4a0e8726a97340dd555b" candidate_uuid: "fadc366fe3d44e068aee86188773c207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1ab1de193eb40a9bb803104ee08c680" is_pre_election: true
I20260928 22:49:25.013731 24173 raft_consensus.cc:2454] T 7ddbef8a3bcc4a0e8726a97340dd555b P e1ab1de193eb40a9bb803104ee08c680 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fadc366fe3d44e068aee86188773c207 in term 0.
I20260928 22:49:25.013765 23836 leader_election.cc:302] T 7ddbef8a3bcc4a0e8726a97340dd555b P fadc366fe3d44e068aee86188773c207 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c12e3d60fcc24875801961d9ddb3fe63, fadc366fe3d44e068aee86188773c207; no voters:
I20260928 22:49:25.013932 24317 raft_consensus.cc:2781] T 7ddbef8a3bcc4a0e8726a97340dd555b P fadc366fe3d44e068aee86188773c207 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:49:25.013990 24317 raft_consensus.cc:482] T 7ddbef8a3bcc4a0e8726a97340dd555b P fadc366fe3d44e068aee86188773c207 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:49:25.014011 24317 raft_consensus.cc:3037] T 7ddbef8a3bcc4a0e8726a97340dd555b P fadc366fe3d44e068aee86188773c207 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:25.014685 24317 raft_consensus.cc:504] T 7ddbef8a3bcc4a0e8726a97340dd555b P fadc366fe3d44e068aee86188773c207 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1ab1de193eb40a9bb803104ee08c680" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36269 } } peers { permanent_uuid: "fadc366fe3d44e068aee86188773c207" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 42243 } } peers { permanent_uuid: "c12e3d60fcc24875801961d9ddb3fe63" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 43775 } }
I20260928 22:49:25.014834 24317 leader_election.cc:288] T 7ddbef8a3bcc4a0e8726a97340dd555b P fadc366fe3d44e068aee86188773c207 [CANDIDATE]: Term 1 election: Requested vote from peers e1ab1de193eb40a9bb803104ee08c680 (127.16.121.131:36269), c12e3d60fcc24875801961d9ddb3fe63 (127.16.121.130:43775)
I20260928 22:49:25.014999 24173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ddbef8a3bcc4a0e8726a97340dd555b" candidate_uuid: "fadc366fe3d44e068aee86188773c207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1ab1de193eb40a9bb803104ee08c680"
I20260928 22:49:25.015116 24173 raft_consensus.cc:3037] T 7ddbef8a3bcc4a0e8726a97340dd555b P e1ab1de193eb40a9bb803104ee08c680 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:25.015084 24037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ddbef8a3bcc4a0e8726a97340dd555b" candidate_uuid: "fadc366fe3d44e068aee86188773c207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12e3d60fcc24875801961d9ddb3fe63"
I20260928 22:49:25.015156 24037 raft_consensus.cc:3037] T 7ddbef8a3bcc4a0e8726a97340dd555b P c12e3d60fcc24875801961d9ddb3fe63 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:25.015977 24037 raft_consensus.cc:2454] T 7ddbef8a3bcc4a0e8726a97340dd555b P c12e3d60fcc24875801961d9ddb3fe63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fadc366fe3d44e068aee86188773c207 in term 1.
I20260928 22:49:25.015977 24173 raft_consensus.cc:2454] T 7ddbef8a3bcc4a0e8726a97340dd555b P e1ab1de193eb40a9bb803104ee08c680 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fadc366fe3d44e068aee86188773c207 in term 1.
I20260928 22:49:25.016191 23836 leader_election.cc:302] T 7ddbef8a3bcc4a0e8726a97340dd555b P fadc366fe3d44e068aee86188773c207 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c12e3d60fcc24875801961d9ddb3fe63, fadc366fe3d44e068aee86188773c207; no voters:
I20260928 22:49:25.016350 24317 raft_consensus.cc:2781] T 7ddbef8a3bcc4a0e8726a97340dd555b P fadc366fe3d44e068aee86188773c207 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:49:25.016530 24317 raft_consensus.cc:686] T 7ddbef8a3bcc4a0e8726a97340dd555b P fadc366fe3d44e068aee86188773c207 [term 1 LEADER]: Becoming Leader. State: Replica: fadc366fe3d44e068aee86188773c207, State: Running, Role: LEADER
I20260928 22:49:25.016620 24317 consensus_queue.cc:237] T 7ddbef8a3bcc4a0e8726a97340dd555b P fadc366fe3d44e068aee86188773c207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1ab1de193eb40a9bb803104ee08c680" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36269 } } peers { permanent_uuid: "fadc366fe3d44e068aee86188773c207" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 42243 } } peers { permanent_uuid: "c12e3d60fcc24875801961d9ddb3fe63" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 43775 } }
I20260928 22:49:25.016929 24322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:25.016811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:39331 (local address 127.16.121.129:34499)
0928 22:49:25.016869 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:25.016899 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:39331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 22:49:25.017580 23579 catalog_manager.cc:5495] T 7ddbef8a3bcc4a0e8726a97340dd555b P fadc366fe3d44e068aee86188773c207 reported cstate change: term changed from 0 to 1, leader changed from <none> to fadc366fe3d44e068aee86188773c207 (127.16.121.129). New cstate: current_term: 1 leader_uuid: "fadc366fe3d44e068aee86188773c207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1ab1de193eb40a9bb803104ee08c680" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fadc366fe3d44e068aee86188773c207" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 42243 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c12e3d60fcc24875801961d9ddb3fe63" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 43775 } health_report { overall_health: UNKNOWN } } }
I20260928 22:49:25.043507 24308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:25.040826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41428 (local address 127.16.121.129:42243)
0928 22:49:25.040917 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:25.040919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:25.040929 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:49:25.041002 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:25.041003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:25.041027 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:25.041096 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:49:25.041227 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:25.041719 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:25.042568 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:25.042764 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:25.042822 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:25.043021 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:49:25.043206 (+ 185us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:49:25.043268 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:25.043368 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 22:49:25.043390 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 22:49:25.044909 24037 raft_consensus.cc:1261] T 7ddbef8a3bcc4a0e8726a97340dd555b P c12e3d60fcc24875801961d9ddb3fe63 [term 1 FOLLOWER]: Refusing update from remote peer fadc366fe3d44e068aee86188773c207: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:49:25.044932 24173 raft_consensus.cc:1261] T 7ddbef8a3bcc4a0e8726a97340dd555b P e1ab1de193eb40a9bb803104ee08c680 [term 1 FOLLOWER]: Refusing update from remote peer fadc366fe3d44e068aee86188773c207: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:49:25.045244 24323 consensus_queue.cc:1035] T 7ddbef8a3bcc4a0e8726a97340dd555b P fadc366fe3d44e068aee86188773c207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12e3d60fcc24875801961d9ddb3fe63" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 43775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:49:25.045392 24317 consensus_queue.cc:1035] T 7ddbef8a3bcc4a0e8726a97340dd555b P fadc366fe3d44e068aee86188773c207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1ab1de193eb40a9bb803104ee08c680" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:49:25.046221 24321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:25.046108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:39331 (local address 127.16.121.130:60179)
0928 22:49:25.046148 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:25.046184 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:39331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260928 22:49:25.046255 24316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:25.046136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:39331 (local address 127.16.121.131:38525)
0928 22:49:25.046180 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:25.046217 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:39331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 22:49:25.047852 24322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:25.047622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:39331 (local address 127.16.121.129:45859)
0928 22:49:25.047777 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:25.047814 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:39331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260928 22:49:25.048382 24316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:25.048243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:39331 (local address 127.16.121.131:56285)
0928 22:49:25.048302 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:25.048355 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:39331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 22:49:25.048671 24321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:25.048568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:39331 (local address 127.16.121.130:42283)
0928 22:49:25.048610 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:25.048635 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:39331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 22:49:25.051340 23578 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:49:25.051462 23578 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:49:25.053022 23578 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a5fee57a42a14e1ab90bc3db2e453559: Sending DeleteTablet for 3 replicas of tablet 7ddbef8a3bcc4a0e8726a97340dd555b
I20260928 22:49:25.053392 24153 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ddbef8a3bcc4a0e8726a97340dd555b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:49:25 UTC) from {username='oryx', principal='oryx/127.16.121.189@KRBTEST.COM'} at 127.0.0.1:40924
I20260928 22:49:25.053516 23880 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ddbef8a3bcc4a0e8726a97340dd555b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:49:25 UTC) from {username='oryx', principal='oryx/127.16.121.189@KRBTEST.COM'} at 127.0.0.1:41414
I20260928 22:49:25.053658 24017 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ddbef8a3bcc4a0e8726a97340dd555b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:49:25 UTC) from {username='oryx', principal='oryx/127.16.121.189@KRBTEST.COM'} at 127.0.0.1:42836
I20260928 22:49:25.053725 24336 tablet_replica.cc:330] T 7ddbef8a3bcc4a0e8726a97340dd555b P e1ab1de193eb40a9bb803104ee08c680: stopping tablet replica
I20260928 22:49:25.053808 24336 raft_consensus.cc:2229] T 7ddbef8a3bcc4a0e8726a97340dd555b P e1ab1de193eb40a9bb803104ee08c680 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:49:25.053861 24336 raft_consensus.cc:2258] T 7ddbef8a3bcc4a0e8726a97340dd555b P e1ab1de193eb40a9bb803104ee08c680 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:49:25.054219 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 23726
I20260928 22:49:25.054227 24336 ts_tablet_manager.cc:1905] T 7ddbef8a3bcc4a0e8726a97340dd555b P e1ab1de193eb40a9bb803104ee08c680: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:49:25.055948 24338 tablet_replica.cc:330] T 7ddbef8a3bcc4a0e8726a97340dd555b P c12e3d60fcc24875801961d9ddb3fe63: stopping tablet replica
I20260928 22:49:25.056099 24338 raft_consensus.cc:2229] T 7ddbef8a3bcc4a0e8726a97340dd555b P c12e3d60fcc24875801961d9ddb3fe63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:49:25.056207 24338 raft_consensus.cc:2258] T 7ddbef8a3bcc4a0e8726a97340dd555b P c12e3d60fcc24875801961d9ddb3fe63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:49:25.057104 24336 ts_tablet_manager.cc:1918] T 7ddbef8a3bcc4a0e8726a97340dd555b P e1ab1de193eb40a9bb803104ee08c680: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:49:25.057212 24336 log.cc:1196] T 7ddbef8a3bcc4a0e8726a97340dd555b P e1ab1de193eb40a9bb803104ee08c680: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-2/wal/wals/7ddbef8a3bcc4a0e8726a97340dd555b
I20260928 22:49:25.057466 24338 ts_tablet_manager.cc:1905] T 7ddbef8a3bcc4a0e8726a97340dd555b P c12e3d60fcc24875801961d9ddb3fe63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:49:25.057648 24336 ts_tablet_manager.cc:1939] T 7ddbef8a3bcc4a0e8726a97340dd555b P e1ab1de193eb40a9bb803104ee08c680: Deleting consensus metadata
I20260928 22:49:25.058090 23564 catalog_manager.cc:4841] TS e1ab1de193eb40a9bb803104ee08c680 (127.16.121.131:36269): tablet 7ddbef8a3bcc4a0e8726a97340dd555b (table test-table [id=0693468d6d064a47a2d84ffb0eb16b62]) successfully deleted
I20260928 22:49:25.059185 24338 ts_tablet_manager.cc:1918] T 7ddbef8a3bcc4a0e8726a97340dd555b P c12e3d60fcc24875801961d9ddb3fe63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:49:25.059237 24338 log.cc:1196] T 7ddbef8a3bcc4a0e8726a97340dd555b P c12e3d60fcc24875801961d9ddb3fe63: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790635733329745-16870-0/minicluster-data/ts-1/wal/wals/7ddbef8a3bcc4a0e8726a97340dd555b
I20260928 22:49:25.059567 24338 ts_tablet_manager.cc:1939] T 7ddbef8a3bcc4a0e8726a97340dd555b P c12e3d60fcc24875801961d9ddb3fe63: Deleting consensus metadata
I20260928 22:49:25.059962 23566 catalog_manager.cc:4841] TS c12e3d60fcc24875801961d9ddb3fe63 (127.16.121.130:43775): tablet 7ddbef8a3bcc4a0e8726a97340dd555b (table test-table [id=0693468d6d064a47a2d84ffb0eb16b62]) successfully deleted
W20260928 22:49:25.061439 24305 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:25.061273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:42243 (local address 127.0.0.1:41444)
0928 22:49:25.061336 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:25.061367 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.129:42243: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 22:49:25.061633 23566 catalog_manager.cc:4584] TS fadc366fe3d44e068aee86188773c207 (127.16.121.129:42243): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7ddbef8a3bcc4a0e8726a97340dd555b: Network error: Client connection negotiation failed: client connection to 127.16.121.129:42243: connect: Connection reset by peer (error 104)
I20260928 22:49:25.061697 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 23954
I20260928 22:49:25.067621 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 24090
I20260928 22:49:25.073933 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 23446
I20260928 22:49:25.079330 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 23633
2026-09-28T22:49:25Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4863 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:49:25 dist-test-slave-56vw krb5kdc[24341](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:49:25 dist-test-slave-56vw krb5kdc[24341](info): set up 2 sockets
Sep 28 22:49:25 dist-test-slave-56vw krb5kdc[24341](info): commencing operation
krb5kdc: starting...
W20260928 22:49:26.199848 16870 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Sep 28 22:49:26 dist-test-slave-56vw krb5kdc[24341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:49:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:49:26Z Disabled control of system clock
WARNING: no policy specified for oryx/127.16.121.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.121.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.121.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.190.keytab.
Entry for principal oryx/127.16.121.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.190.keytab.
I20260928 22:49:26.331480 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:40321
--webserver_interface=127.16.121.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.190.keytab
--principal=oryx/127.16.121.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:42859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:40321
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:49:26.433909 24354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:26.434178 24354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:26.434226 24354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:26.435801 24354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:49:26.435911 24354 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 22:49:26.435940 24354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:26.435974 24354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:26.436005 24354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:49:26.436033 24354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:49:26.438225 24354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:42859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:40321
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.190.keytab
--principal=oryx/127.16.121.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:40321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.121.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:26.438591 24354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:26.438858 24354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:26.445559 24359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:26.445705 24354 server_base.cc:1034] running on GCE node
W20260928 22:49:26.445562 24362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:26.445562 24360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:26.446169 24354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:26.446596 24354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:26.447870 24354 hybrid_clock.cc:648] HybridClock initialized: now 1790635766447706 us; error 107 us; skew 500 ppm
Sep 28 22:49:26 dist-test-slave-56vw krb5kdc[24341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635766, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.121.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:26.451867 24354 init.cc:377] Logged in from keytab as oryx/127.16.121.190@KRBTEST.COM (short username oryx)
I20260928 22:49:26.452418 24354 webserver.cc:460] Webserver started at http://127.16.121.190:32903/ using document root <none> and password file <none>
I20260928 22:49:26.452702 24354 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:26.452775 24354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:26.453154 24354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:26.454404 24354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "20a5b16187d042b5ac6e0609f080bced"
format_stamp: "Formatted at 2026-09-28 22:49:26 on dist-test-slave-56vw"
server_key: "c2a51d3d3ee3a7bb81e3d96cc1afb701"
server_key_iv: "2bf25fdfbbe9daf308296a6b755d6d17"
server_key_version: "encryptionkey@0"
I20260928 22:49:26.454787 24354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "20a5b16187d042b5ac6e0609f080bced"
format_stamp: "Formatted at 2026-09-28 22:49:26 on dist-test-slave-56vw"
server_key: "c2a51d3d3ee3a7bb81e3d96cc1afb701"
server_key_iv: "2bf25fdfbbe9daf308296a6b755d6d17"
server_key_version: "encryptionkey@0"
I20260928 22:49:26.456660 24354 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260928 22:49:26.457722 24369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:26.458076 24354 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:49:26.458302 24354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "20a5b16187d042b5ac6e0609f080bced"
format_stamp: "Formatted at 2026-09-28 22:49:26 on dist-test-slave-56vw"
server_key: "c2a51d3d3ee3a7bb81e3d96cc1afb701"
server_key_iv: "2bf25fdfbbe9daf308296a6b755d6d17"
server_key_version: "encryptionkey@0"
I20260928 22:49:26.458396 24354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:26.501750 24354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:26.504483 24354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:26.504673 24354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:26.510814 24354 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.190:40321
I20260928 22:49:26.510931 24431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.190:40321 every 8 connection(s)
I20260928 22:49:26.511319 24354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:49:26.512122 24432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:26.514477 24432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20a5b16187d042b5ac6e0609f080bced: Bootstrap starting.
I20260928 22:49:26.515084 24432 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20a5b16187d042b5ac6e0609f080bced: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:26.515336 24432 log.cc:824] T 00000000000000000000000000000000 P 20a5b16187d042b5ac6e0609f080bced: Log is configured to *not* fsync() on all Append() calls
I20260928 22:49:26.516059 24432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20a5b16187d042b5ac6e0609f080bced: No bootstrap required, opened a new log
I20260928 22:49:26.518206 24432 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20a5b16187d042b5ac6e0609f080bced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20a5b16187d042b5ac6e0609f080bced" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 40321 } }
I20260928 22:49:26.518365 24432 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20a5b16187d042b5ac6e0609f080bced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:26.518397 24432 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20a5b16187d042b5ac6e0609f080bced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a5b16187d042b5ac6e0609f080bced, State: Initialized, Role: FOLLOWER
I20260928 22:49:26.518564 24432 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20a5b16187d042b5ac6e0609f080bced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20a5b16187d042b5ac6e0609f080bced" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 40321 } }
I20260928 22:49:26.518664 24432 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20a5b16187d042b5ac6e0609f080bced [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:49:26.518690 24432 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20a5b16187d042b5ac6e0609f080bced [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:49:26.518720 24432 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20a5b16187d042b5ac6e0609f080bced [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:26.519341 24432 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20a5b16187d042b5ac6e0609f080bced [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20a5b16187d042b5ac6e0609f080bced" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 40321 } }
I20260928 22:49:26.519447 24432 leader_election.cc:302] T 00000000000000000000000000000000 P 20a5b16187d042b5ac6e0609f080bced [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20a5b16187d042b5ac6e0609f080bced; no voters:
I20260928 22:49:26.519590 24432 leader_election.cc:288] T 00000000000000000000000000000000 P 20a5b16187d042b5ac6e0609f080bced [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:49:26.519693 24435 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20a5b16187d042b5ac6e0609f080bced [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:49:26.519868 24435 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20a5b16187d042b5ac6e0609f080bced [term 1 LEADER]: Becoming Leader. State: Replica: 20a5b16187d042b5ac6e0609f080bced, State: Running, Role: LEADER
I20260928 22:49:26.519889 24432 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20a5b16187d042b5ac6e0609f080bced [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:49:26.519964 24435 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20a5b16187d042b5ac6e0609f080bced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20a5b16187d042b5ac6e0609f080bced" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 40321 } }
I20260928 22:49:26.520356 24436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20a5b16187d042b5ac6e0609f080bced [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20a5b16187d042b5ac6e0609f080bced" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20a5b16187d042b5ac6e0609f080bced" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 40321 } } }
I20260928 22:49:26.520345 24435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20a5b16187d042b5ac6e0609f080bced [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20a5b16187d042b5ac6e0609f080bced. Latest consensus state: current_term: 1 leader_uuid: "20a5b16187d042b5ac6e0609f080bced" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20a5b16187d042b5ac6e0609f080bced" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 40321 } } }
I20260928 22:49:26.520541 24436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20a5b16187d042b5ac6e0609f080bced [sys.catalog]: This master's current role is: LEADER
I20260928 22:49:26.520553 24435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20a5b16187d042b5ac6e0609f080bced [sys.catalog]: This master's current role is: LEADER
I20260928 22:49:26.521093 24444 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:49:26.521226 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 24354
I20260928 22:49:26.521311 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:49:26.521538 16870 external_mini_cluster.cc:1428] Setting key e88f371714c98d91abc9f346eb859d2b
I20260928 22:49:26.523285 24444 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:49:26.525048 24444 catalog_manager.cc:1347] Generated new cluster ID: 005e666034874fdcaf74101f1ad79e0f
I20260928 22:49:26.525105 24444 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 22:49:26 dist-test-slave-56vw krb5kdc[24341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM
I20260928 22:49:26.533821 24453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:26.522278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44102 (local address 127.16.121.190:40321)
0928 22:49:26.522479 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:26.522490 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:26.522522 (+ 32us) server_negotiation.cc:407] Connection header received
0928 22:49:26.522651 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:26.522659 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:26.522717 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:26.522869 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:26.523723 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:26.525069 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:26.526237 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:26.526698 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:26.526834 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:26.529801 (+ 2967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:26.529808 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:26.529808 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:26.529825 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:26.532069 (+ 2244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:26.532505 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:26.532509 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:26.532509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:26.532618 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:26.532837 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:26.532839 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:26.532839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:26.533130 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:26.533306 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:26.533415 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 22:49:26.533574 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":104,"threads_started":1}
I20260928 22:49:26.548668 24444 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:49:26.549381 24444 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:49:26.582219 24444 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20a5b16187d042b5ac6e0609f080bced: Generated new TSK 0
I20260928 22:49:26.582551 24444 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.16.121.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.121.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.121.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.129.keytab.
Entry for principal oryx/127.16.121.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.129.keytab.
I20260928 22:49:26.627761 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.129:0
--local_ip_for_outbound_sockets=127.16.121.129
--webserver_interface=127.16.121.129
--webserver_port=0
--tserver_master_addrs=127.16.121.190:40321
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.129.keytab
--principal=oryx/127.16.121.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:42859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:49:26.733604 24458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:26.733842 24458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:26.733884 24458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:26.735194 24458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:26.735244 24458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:26.735267 24458 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 22:49:26.735286 24458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.129
I20260928 22:49:26.736929 24458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:42859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.129.keytab
--principal=oryx/127.16.121.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.121.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.121.190:40321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.129
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:26.737169 24458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:26.737371 24458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:26.743427 24464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:26.743427 24466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:26.743449 24463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:26.743876 24458 server_base.cc:1034] running on GCE node
I20260928 22:49:26.744130 24458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:26.744424 24458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:26.745594 24458 hybrid_clock.cc:648] HybridClock initialized: now 1790635766745581 us; error 32 us; skew 500 ppm
Sep 28 22:49:26 dist-test-slave-56vw krb5kdc[24341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635766, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.121.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:26.748502 24458 init.cc:377] Logged in from keytab as oryx/127.16.121.129@KRBTEST.COM (short username oryx)
I20260928 22:49:26.749080 24458 webserver.cc:460] Webserver started at http://127.16.121.129:36363/ using document root <none> and password file <none>
I20260928 22:49:26.749360 24458 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:26.749440 24458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:26.749609 24458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:26.750870 24458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-0/data/instance:
uuid: "33643294a0eb433e8931f67583d665e8"
format_stamp: "Formatted at 2026-09-28 22:49:26 on dist-test-slave-56vw"
server_key: "8489441e7757c4963f8d4505e06b6ce5"
server_key_iv: "e487bcc0fa25e8cd732bdc5a40f0bf54"
server_key_version: "encryptionkey@0"
I20260928 22:49:26.751328 24458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance:
uuid: "33643294a0eb433e8931f67583d665e8"
format_stamp: "Formatted at 2026-09-28 22:49:26 on dist-test-slave-56vw"
server_key: "8489441e7757c4963f8d4505e06b6ce5"
server_key_iv: "e487bcc0fa25e8cd732bdc5a40f0bf54"
server_key_version: "encryptionkey@0"
I20260928 22:49:26.753022 24458 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:49:26.754042 24473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:26.754458 24458 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260928 22:49:26.754628 24458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-0/wal
uuid: "33643294a0eb433e8931f67583d665e8"
format_stamp: "Formatted at 2026-09-28 22:49:26 on dist-test-slave-56vw"
server_key: "8489441e7757c4963f8d4505e06b6ce5"
server_key_iv: "e487bcc0fa25e8cd732bdc5a40f0bf54"
server_key_version: "encryptionkey@0"
I20260928 22:49:26.754746 24458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:26.770233 24458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:26.772797 24458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:26.773114 24458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:26.773659 24458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:49:26.773722 24458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:26.773774 24458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:49:26.773808 24458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 22:49:26 dist-test-slave-56vw krb5kdc[24341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635766, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.121.129@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM
I20260928 22:49:26.781612 24458 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.129:37109
I20260928 22:49:26.781649 24588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.129:37109 every 8 connection(s)
I20260928 22:49:26.782090 24458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
I20260928 22:49:26.784266 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 24458
I20260928 22:49:26.784443 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance
I20260928 22:49:26.784691 16870 external_mini_cluster.cc:1428] Setting key aea36e345d7deebc15a76f2fca4146cf
I20260928 22:49:26.785745 24453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:26.774301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:56117 (local address 127.16.121.190:40321)
0928 22:49:26.774381 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:26.774384 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:26.774741 (+ 357us) server_negotiation.cc:407] Connection header received
0928 22:49:26.775530 (+ 789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:26.775532 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:26.775551 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:26.775612 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:26.776672 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:26.777631 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:26.778874 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:26.779164 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:26.779289 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:26.782112 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:26.782114 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:26.782115 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:26.782128 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:26.783829 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:26.784552 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:26.784553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:26.784554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:26.784618 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:26.784990 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:26.784991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:26.784991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:26.785122 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:26.785215 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:26.785542 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 22:49:26.785642 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 22:49:26.785974 24482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:26.774279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:40321 (local address 127.16.121.129:56117)
0928 22:49:26.774573 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:26.774620 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:26.775398 (+ 778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:26.775728 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:26.775733 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:26.776185 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:26.776538 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:26.776545 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:26.777751 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:26.777773 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:26.778732 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:26.778735 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:26.779310 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:26.779312 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:26.779468 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:26.780114 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:26.780124 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:26.781986 (+ 1862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:26.784007 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:26.784014 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:26.784016 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:26.784275 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:26.784701 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:26.784702 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:26.784703 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:26.784851 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:26.785205 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:26.785212 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:26.785387 (+ 175us) client_negotiation.cc:769] Sending connection context
0928 22:49:26.785517 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 22:49:26.785711 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":117,"threads_started":1}
I20260928 22:49:26.787511 24589 heartbeater.cc:347] Connected to a master server at 127.16.121.190:40321
I20260928 22:49:26.787627 24589 heartbeater.cc:464] Registering TS with master...
I20260928 22:49:26.787828 24589 heartbeater.cc:511] Master 127.16.121.190:40321 requested a full tablet report, sending...
I20260928 22:49:26.788461 24385 ts_manager.cc:194] Registered new tserver with Master: 33643294a0eb433e8931f67583d665e8 (127.16.121.129:37109)
I20260928 22:49:26.789196 24385 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.121.129@KRBTEST.COM'} at 127.16.121.129:56117
WARNING: no policy specified for oryx/127.16.121.130@KRBTEST.COM; defaulting to no policy
I20260928 22:49:26.795354 24453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:26.788737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:53669 (local address 127.16.121.190:40321)
0928 22:49:26.789404 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:26.789406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:26.789426 (+ 20us) server_negotiation.cc:407] Connection header received
0928 22:49:26.789447 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:26.789447 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:26.789456 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:26.789527 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:26.789978 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:26.790476 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:26.791251 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:26.791463 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:26.791551 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:26.792589 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:26.792591 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:26.792592 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:26.792600 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:26.794067 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:26.794506 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:26.794507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:26.794508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:26.794574 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:26.794782 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:26.794783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:26.794783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:26.794908 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:26.794986 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:26.795166 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 22:49:26.795255 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643}
I20260928 22:49:26.795388 24596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:26.788567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:40321 (local address 127.16.121.129:53669)
0928 22:49:26.788713 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:26.788726 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:26.788785 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:26.789548 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:26.789550 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:26.789817 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:26.789886 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:26.789888 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:26.790574 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:26.790575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:26.791159 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:26.791161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:26.791555 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:26.791556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:26.791626 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:26.792161 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:26.792166 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:26.792495 (+ 329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:26.794176 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:26.794177 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:26.794177 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:26.794410 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:26.794641 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:26.794642 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:26.794643 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:26.794706 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:26.794991 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:26.794992 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:26.795067 (+ 75us) client_negotiation.cc:769] Sending connection context
0928 22:49:26.795140 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 22:49:26.795255 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":84,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.121.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.121.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.130.keytab.
Entry for principal oryx/127.16.121.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.130.keytab.
I20260928 22:49:26.815322 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.130:0
--local_ip_for_outbound_sockets=127.16.121.130
--webserver_interface=127.16.121.130
--webserver_port=0
--tserver_master_addrs=127.16.121.190:40321
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.130.keytab
--principal=oryx/127.16.121.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:42859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:49:26.903512 24598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:26.903743 24598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:26.903769 24598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:26.905058 24598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:26.905115 24598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:26.905139 24598 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 22:49:26.905158 24598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.130
I20260928 22:49:26.906910 24598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:42859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.130.keytab
--principal=oryx/127.16.121.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.121.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.121.190:40321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.130
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:26.907212 24598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:26.907507 24598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:26.912813 24603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:26.912839 24606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:26.912834 24604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:26.913472 24598 server_base.cc:1034] running on GCE node
I20260928 22:49:26.913738 24598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:26.914026 24598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:26.915177 24598 hybrid_clock.cc:648] HybridClock initialized: now 1790635766915100 us; error 93 us; skew 500 ppm
Sep 28 22:49:26 dist-test-slave-56vw krb5kdc[24341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635766, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.121.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:26.918105 24598 init.cc:377] Logged in from keytab as oryx/127.16.121.130@KRBTEST.COM (short username oryx)
I20260928 22:49:26.918793 24598 webserver.cc:460] Webserver started at http://127.16.121.130:40497/ using document root <none> and password file <none>
I20260928 22:49:26.919111 24598 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:26.919193 24598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:26.919374 24598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:26.920800 24598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-1/data/instance:
uuid: "df6cca547d3847d08cf92cec310077c8"
format_stamp: "Formatted at 2026-09-28 22:49:26 on dist-test-slave-56vw"
server_key: "02127dd07f9013704219eae70822326d"
server_key_iv: "14559c1f43fd2bf1645d7fcc88151604"
server_key_version: "encryptionkey@0"
I20260928 22:49:26.921240 24598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance:
uuid: "df6cca547d3847d08cf92cec310077c8"
format_stamp: "Formatted at 2026-09-28 22:49:26 on dist-test-slave-56vw"
server_key: "02127dd07f9013704219eae70822326d"
server_key_iv: "14559c1f43fd2bf1645d7fcc88151604"
server_key_version: "encryptionkey@0"
I20260928 22:49:26.922822 24598 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:49:26.924007 24613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:26.924288 24598 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:49:26.924512 24598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-1/wal
uuid: "df6cca547d3847d08cf92cec310077c8"
format_stamp: "Formatted at 2026-09-28 22:49:26 on dist-test-slave-56vw"
server_key: "02127dd07f9013704219eae70822326d"
server_key_iv: "14559c1f43fd2bf1645d7fcc88151604"
server_key_version: "encryptionkey@0"
I20260928 22:49:26.924612 24598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:26.944306 24598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:26.947069 24598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:26.947270 24598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:26.947926 24598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:49:26.947978 24598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:26.948010 24598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:49:26.948031 24598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 22:49:26 dist-test-slave-56vw krb5kdc[24341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635766, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.121.130@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM
I20260928 22:49:26.955592 24598 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.130:38577
I20260928 22:49:26.955951 24598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
I20260928 22:49:26.956107 24728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.130:38577 every 8 connection(s)
I20260928 22:49:26.959412 24453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:26.948423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:49839 (local address 127.16.121.190:40321)
0928 22:49:26.948494 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:26.948497 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:26.948728 (+ 231us) server_negotiation.cc:407] Connection header received
0928 22:49:26.949259 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:26.949261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:26.949277 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:26.949418 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:26.950129 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:26.950943 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:26.952746 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:26.953030 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:26.953126 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:26.956028 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:26.956030 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:26.956031 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:26.956040 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:26.957637 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:26.958301 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:26.958303 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:26.958303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:26.958369 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:26.958652 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:26.958653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:26.958654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:26.958789 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:26.958865 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:26.959229 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 22:49:26.959317 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 22:49:26.959640 24622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:26.948406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:40321 (local address 127.16.121.130:49839)
0928 22:49:26.948582 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:26.948627 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:26.949153 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:26.949494 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:26.949498 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:26.949805 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:26.950022 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:26.950026 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:26.951040 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:26.951041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:26.952630 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:26.952633 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:26.953149 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:26.953150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:26.953287 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:26.954293 (+ 1006us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:26.954301 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:26.955915 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:26.957747 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:26.957754 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:26.957756 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:26.958189 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:26.958444 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:26.958445 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:26.958445 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:26.958578 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:26.958878 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:26.958885 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:26.959033 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 22:49:26.959206 (+ 173us) client_negotiation.cc:240] Negotiation successful
0928 22:49:26.959382 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":100,"threads_started":1}
I20260928 22:49:26.960404 24729 heartbeater.cc:347] Connected to a master server at 127.16.121.190:40321
I20260928 22:49:26.960510 24729 heartbeater.cc:464] Registering TS with master...
I20260928 22:49:26.960727 24729 heartbeater.cc:511] Master 127.16.121.190:40321 requested a full tablet report, sending...
I20260928 22:49:26.961158 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 24598
I20260928 22:49:26.961248 24385 ts_manager.cc:194] Registered new tserver with Master: df6cca547d3847d08cf92cec310077c8 (127.16.121.130:38577)
I20260928 22:49:26.961287 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance
I20260928 22:49:26.961500 16870 external_mini_cluster.cc:1428] Setting key 283857fa55ba395a6833c0cd22081847
I20260928 22:49:26.961740 24385 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.121.130@KRBTEST.COM'} at 127.16.121.130:49839
I20260928 22:49:26.969309 24453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:26.962083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:33845 (local address 127.16.121.190:40321)
0928 22:49:26.962182 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:26.962184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:26.962190 (+ 6us) server_negotiation.cc:407] Connection header received
0928 22:49:26.962210 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:26.962211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:26.962220 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:26.962283 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:26.962691 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:26.963629 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:26.964838 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:26.965040 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:26.965103 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:26.966336 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:26.966337 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:26.966337 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:26.966345 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:26.967948 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:26.968399 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:26.968400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:26.968400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:26.968457 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:26.968720 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:26.968721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:26.968722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:26.968900 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:26.968966 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:26.969156 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 22:49:26.969225 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 22:49:26.969364 24735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:26.961827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:40321 (local address 127.16.121.130:33845)
0928 22:49:26.961997 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:26.962009 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:26.962059 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:26.962334 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:26.962335 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:26.962504 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:26.962615 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:26.962616 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:26.963782 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:26.963783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:26.964663 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:26.964666 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:26.965108 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:26.965109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:26.965187 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:26.965765 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:26.965771 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:26.966227 (+ 456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:26.968040 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:26.968041 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:26.968042 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:26.968301 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:26.968537 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:26.968538 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:26.968539 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:26.968620 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:26.968983 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:26.968984 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:26.969043 (+ 59us) client_negotiation.cc:769] Sending connection context
0928 22:49:26.969129 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 22:49:26.969251 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":69,"threads_started":1}
WARNING: no policy specified for oryx/127.16.121.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.121.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.121.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.131.keytab.
Entry for principal oryx/127.16.121.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.131.keytab.
I20260928 22:49:26.996515 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.131:0
--local_ip_for_outbound_sockets=127.16.121.131
--webserver_interface=127.16.121.131
--webserver_port=0
--tserver_master_addrs=127.16.121.190:40321
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.131.keytab
--principal=oryx/127.16.121.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:42859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:49:27.091192 24738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:27.091411 24738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:27.091439 24738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:27.092721 24738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:27.092769 24738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:27.092792 24738 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 22:49:27.092809 24738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.131
I20260928 22:49:27.094473 24738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:42859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.131.keytab
--principal=oryx/127.16.121.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.121.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.121.190:40321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.131
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:27.094714 24738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:27.094940 24738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:27.100988 24746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:27.100979 24744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:27.101150 24738 server_base.cc:1034] running on GCE node
W20260928 22:49:27.100979 24743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:27.101603 24738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:27.101922 24738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:27.103144 24738 hybrid_clock.cc:648] HybridClock initialized: now 1790635767103122 us; error 35 us; skew 500 ppm
Sep 28 22:49:27 dist-test-slave-56vw krb5kdc[24341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635767, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.121.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:27.106137 24738 init.cc:377] Logged in from keytab as oryx/127.16.121.131@KRBTEST.COM (short username oryx)
I20260928 22:49:27.106673 24738 webserver.cc:460] Webserver started at http://127.16.121.131:35323/ using document root <none> and password file <none>
I20260928 22:49:27.106971 24738 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:27.107046 24738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:27.107224 24738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:27.108834 24738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-2/data/instance:
uuid: "82c52234a3a94312aae42cfb5ab3ff58"
format_stamp: "Formatted at 2026-09-28 22:49:27 on dist-test-slave-56vw"
server_key: "b95616a8c484a9e33c0ebbc2cf0b0d07"
server_key_iv: "9f5fc86a0221025e9572416f69794290"
server_key_version: "encryptionkey@0"
I20260928 22:49:27.109293 24738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance:
uuid: "82c52234a3a94312aae42cfb5ab3ff58"
format_stamp: "Formatted at 2026-09-28 22:49:27 on dist-test-slave-56vw"
server_key: "b95616a8c484a9e33c0ebbc2cf0b0d07"
server_key_iv: "9f5fc86a0221025e9572416f69794290"
server_key_version: "encryptionkey@0"
I20260928 22:49:27.110963 24738 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:49:27.112076 24753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:27.112358 24738 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:49:27.112506 24738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-2/wal
uuid: "82c52234a3a94312aae42cfb5ab3ff58"
format_stamp: "Formatted at 2026-09-28 22:49:27 on dist-test-slave-56vw"
server_key: "b95616a8c484a9e33c0ebbc2cf0b0d07"
server_key_iv: "9f5fc86a0221025e9572416f69794290"
server_key_version: "encryptionkey@0"
I20260928 22:49:27.112658 24738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:27.131664 24738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:27.134851 24738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:27.135213 24738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:27.135895 24738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:49:27.135967 24738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:27.136008 24738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:49:27.136036 24738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 22:49:27 dist-test-slave-56vw krb5kdc[24341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635767, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.121.131@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM
I20260928 22:49:27.144487 24738 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.131:36097
I20260928 22:49:27.144585 24868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.131:36097 every 8 connection(s)
I20260928 22:49:27.145020 24738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
I20260928 22:49:27.148587 24453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:27.136595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:58219 (local address 127.16.121.190:40321)
0928 22:49:27.136666 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:27.136668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:27.136932 (+ 264us) server_negotiation.cc:407] Connection header received
0928 22:49:27.137729 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:27.137731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:27.137746 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:27.137828 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:27.138914 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.139789 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.141577 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.141914 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.142004 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:27.144883 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:27.144886 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:27.144886 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:27.144898 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:27.146779 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:27.147523 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:27.147524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:27.147525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:27.147603 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:27.147925 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:27.147926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:27.147926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:27.148057 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:27.148122 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:27.148436 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 22:49:27.148518 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 22:49:27.148814 24762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:27.136577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:40321 (local address 127.16.121.131:58219)
0928 22:49:27.136774 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:27.136830 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:27.137601 (+ 771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:27.137973 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:27.137979 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:27.138411 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:27.138757 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:27.138765 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.139918 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.139920 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:27.141355 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:27.141358 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.142035 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.142036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:27.142191 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:27.143016 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:27.143030 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:27.144749 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:27.146896 (+ 2147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:27.146903 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:27.146904 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:27.147283 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:27.147698 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:27.147699 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:27.147700 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:27.147833 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:27.148146 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:27.148151 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:27.148289 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 22:49:27.148397 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 22:49:27.148537 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":85,"threads_started":1}
I20260928 22:49:27.149379 24869 heartbeater.cc:347] Connected to a master server at 127.16.121.190:40321
I20260928 22:49:27.149466 24869 heartbeater.cc:464] Registering TS with master...
I20260928 22:49:27.149685 24869 heartbeater.cc:511] Master 127.16.121.190:40321 requested a full tablet report, sending...
I20260928 22:49:27.150197 24385 ts_manager.cc:194] Registered new tserver with Master: 82c52234a3a94312aae42cfb5ab3ff58 (127.16.121.131:36097)
I20260928 22:49:27.150696 24385 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.121.131@KRBTEST.COM'} at 127.16.121.131:58219
I20260928 22:49:27.153438 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 24738
I20260928 22:49:27.153571 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance
I20260928 22:49:27.153779 16870 external_mini_cluster.cc:1428] Setting key 937c3c82eeae83c9162491e8e521272d
I20260928 22:49:27.155515 16870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:49:27.157521 24453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:27.150734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:46927 (local address 127.16.121.190:40321)
0928 22:49:27.150871 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:27.150873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:27.150881 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:49:27.150904 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:27.150905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:27.150916 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:27.150999 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:27.151570 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.152238 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.153070 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.153461 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.153559 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:27.154796 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:27.154798 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:27.154799 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:27.154806 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:27.156306 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:27.156692 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:27.156694 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:27.156694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:27.156757 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:27.156967 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:27.156969 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:27.156969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:27.157091 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:27.157171 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:27.157334 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 22:49:27.157426 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260928 22:49:27.157585 24875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:27.150585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:40321 (local address 127.16.121.131:46927)
0928 22:49:27.150742 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:27.150755 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:27.150815 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:27.151006 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:27.151007 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:27.151312 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:27.151410 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:27.151413 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.152428 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.152429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:27.152957 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:27.152958 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.153595 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.153596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:27.153675 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:27.154283 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:27.154288 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:27.154679 (+ 391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:27.156414 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:27.156415 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:27.156416 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:27.156603 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:27.156827 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:27.156828 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:27.156829 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:27.156896 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:27.157160 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:27.157161 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:27.157215 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 22:49:27.157309 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 22:49:27.157473 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":84,"threads_started":1}
Sep 28 22:49:27 dist-test-slave-56vw krb5kdc[24341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635767, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 22:49:27 dist-test-slave-56vw krb5kdc[24341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:27 dist-test-slave-56vw krb5kdc[24341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
W20260928 22:49:27.180500 24453 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:27.174408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44110 (local address 127.16.121.190:40321)
0928 22:49:27.174484 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:27.174487 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:27.174496 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:49:27.174528 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:27.174529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:27.174539 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:27.174607 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:27.175262 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.175763 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.176557 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.176756 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.176847 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:27.180382 (+ 3535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44110: BlockingRecv error: recv got EOF from 127.0.0.1:44110 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 22:49:27 dist-test-slave-56vw krb5kdc[24341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635767, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM
I20260928 22:49:27.191610 24453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:27.181567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44112 (local address 127.16.121.190:40321)
0928 22:49:27.181626 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:27.181628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:27.181681 (+ 53us) server_negotiation.cc:407] Connection header received
0928 22:49:27.181751 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:27.181752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:27.181764 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:27.181818 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:27.182507 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.183184 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.183960 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.184177 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.184241 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:27.186901 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:27.186904 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:27.186904 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:27.186916 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:27.189455 (+ 2539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:27.190026 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:27.190028 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:27.190028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:27.190091 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:27.190540 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:27.190541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:27.190541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:27.190680 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:27.190797 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:27.191204 (+ 407us) server_negotiation.cc:299] Negotiation successful
0928 22:49:27.191518 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 22:49:27.194473 24385 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44112:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:49:27.197237 24385 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:49:27.205590 24896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:27.201144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53028 (local address 127.16.121.130:38577)
0928 22:49:27.201442 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:27.201446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:27.201457 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:49:27.201497 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:27.201500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:27.201515 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:27.201598 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:49:27.201821 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.203019 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.204607 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.205136 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.205255 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:27.205328 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:27.205436 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 22:49:27.205478 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":78,"threads_started":1}
I20260928 22:49:27.205583 24893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:27.200806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.130:38577 (local address 127.0.0.1:53028)
0928 22:49:27.201258 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:27.201269 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:27.201313 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:27.201615 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:27.201616 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:27.201618 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:49:27.201659 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:27.201660 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.203155 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.203156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:27.204497 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:27.204499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.205236 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.205237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:27.205323 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:27.205338 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 22:49:27.205418 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 22:49:27.205478 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":218,"thread_start_us":127,"threads_started":1}
I20260928 22:49:27.205955 24892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:27.200550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.131:36097 (local address 127.0.0.1:40098)
0928 22:49:27.200772 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:27.200826 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:27.200912 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:27.201299 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:27.201303 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:27.201308 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:49:27.201406 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:27.201409 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.203098 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.203100 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:27.204481 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:27.204484 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.205497 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.205498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:27.205582 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:27.205588 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 22:49:27.205627 (+ 39us) client_negotiation.cc:240] Negotiation successful
0928 22:49:27.205862 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":195,"thread_start_us":77,"threads_started":1}
I20260928 22:49:27.206202 24891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:27.200549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:37109 (local address 127.0.0.1:52210)
0928 22:49:27.200985 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:27.200994 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:27.201034 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:27.201333 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:27.201333 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:27.201335 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:49:27.201435 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:27.201438 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.202872 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.202873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:27.204295 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:27.204299 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.205936 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.205937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:27.206010 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:27.206014 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:49:27.206081 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 22:49:27.206130 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"mutex_wait_us":61,"thread_start_us":78,"threads_started":1}
I20260928 22:49:27.206486 24894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:27.200852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40098 (local address 127.16.121.131:36097)
0928 22:49:27.201086 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:27.201090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:27.201100 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:49:27.201144 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:27.201147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:27.201164 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:27.201595 (+ 431us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:49:27.201664 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.202954 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.204599 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.205384 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.206182 (+ 798us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:27.206268 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:27.206303 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 22:49:27.206365 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":91,"threads_started":1}
I20260928 22:49:27.206580 24895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:27.200884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52210 (local address 127.16.121.129:37109)
0928 22:49:27.201080 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:27.201087 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:27.201100 (+ 13us) server_negotiation.cc:407] Connection header received
0928 22:49:27.201145 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:27.201147 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:27.201164 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:27.201263 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:49:27.201532 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.202762 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.204424 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.205299 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.205411 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:27.205498 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:27.206398 (+ 900us) server_negotiation.cc:299] Negotiation successful
0928 22:49:27.206452 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":80,"threads_started":1}
I20260928 22:49:27.206504 24663 tablet_service.cc:1467] Processing CreateTablet for tablet d4804397ef3448b8aac0ef154106fc96 (DEFAULT_TABLE table=test-table [id=550e8b7a271046ac9aae6856f523b91b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:49:27.206969 24663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4804397ef3448b8aac0ef154106fc96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:27.207244 24803 tablet_service.cc:1467] Processing CreateTablet for tablet d4804397ef3448b8aac0ef154106fc96 (DEFAULT_TABLE table=test-table [id=550e8b7a271046ac9aae6856f523b91b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:49:27.207612 24803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4804397ef3448b8aac0ef154106fc96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:27.210330 24897 tablet_bootstrap.cc:492] T d4804397ef3448b8aac0ef154106fc96 P df6cca547d3847d08cf92cec310077c8: Bootstrap starting.
I20260928 22:49:27.211023 24898 tablet_bootstrap.cc:492] T d4804397ef3448b8aac0ef154106fc96 P 82c52234a3a94312aae42cfb5ab3ff58: Bootstrap starting.
I20260928 22:49:27.210997 24523 tablet_service.cc:1467] Processing CreateTablet for tablet d4804397ef3448b8aac0ef154106fc96 (DEFAULT_TABLE table=test-table [id=550e8b7a271046ac9aae6856f523b91b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:49:27.211270 24897 tablet_bootstrap.cc:654] T d4804397ef3448b8aac0ef154106fc96 P df6cca547d3847d08cf92cec310077c8: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:27.211423 24523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4804397ef3448b8aac0ef154106fc96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:27.211596 24897 log.cc:824] T d4804397ef3448b8aac0ef154106fc96 P df6cca547d3847d08cf92cec310077c8: Log is configured to *not* fsync() on all Append() calls
I20260928 22:49:27.211694 24898 tablet_bootstrap.cc:654] T d4804397ef3448b8aac0ef154106fc96 P 82c52234a3a94312aae42cfb5ab3ff58: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:27.212095 24898 log.cc:824] T d4804397ef3448b8aac0ef154106fc96 P 82c52234a3a94312aae42cfb5ab3ff58: Log is configured to *not* fsync() on all Append() calls
I20260928 22:49:27.212651 24897 tablet_bootstrap.cc:492] T d4804397ef3448b8aac0ef154106fc96 P df6cca547d3847d08cf92cec310077c8: No bootstrap required, opened a new log
I20260928 22:49:27.212721 24897 ts_tablet_manager.cc:1397] T d4804397ef3448b8aac0ef154106fc96 P df6cca547d3847d08cf92cec310077c8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:49:27.212939 24898 tablet_bootstrap.cc:492] T d4804397ef3448b8aac0ef154106fc96 P 82c52234a3a94312aae42cfb5ab3ff58: No bootstrap required, opened a new log
I20260928 22:49:27.212987 24898 ts_tablet_manager.cc:1397] T d4804397ef3448b8aac0ef154106fc96 P 82c52234a3a94312aae42cfb5ab3ff58: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 22:49:27.214506 24898 raft_consensus.cc:348] T d4804397ef3448b8aac0ef154106fc96 P 82c52234a3a94312aae42cfb5ab3ff58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } } peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } }
I20260928 22:49:27.214661 24898 raft_consensus.cc:374] T d4804397ef3448b8aac0ef154106fc96 P 82c52234a3a94312aae42cfb5ab3ff58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:27.214685 24898 raft_consensus.cc:729] T d4804397ef3448b8aac0ef154106fc96 P 82c52234a3a94312aae42cfb5ab3ff58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82c52234a3a94312aae42cfb5ab3ff58, State: Initialized, Role: FOLLOWER
I20260928 22:49:27.214834 24898 consensus_queue.cc:260] T d4804397ef3448b8aac0ef154106fc96 P 82c52234a3a94312aae42cfb5ab3ff58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } } peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } }
I20260928 22:49:27.215179 24898 ts_tablet_manager.cc:1428] T d4804397ef3448b8aac0ef154106fc96 P 82c52234a3a94312aae42cfb5ab3ff58: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 22:49:27.215094 24897 raft_consensus.cc:348] T d4804397ef3448b8aac0ef154106fc96 P df6cca547d3847d08cf92cec310077c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } } peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } }
I20260928 22:49:27.215359 24897 raft_consensus.cc:374] T d4804397ef3448b8aac0ef154106fc96 P df6cca547d3847d08cf92cec310077c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:27.215380 24869 heartbeater.cc:503] Master 127.16.121.190:40321 was elected leader, sending a full tablet report...
I20260928 22:49:27.215425 24897 raft_consensus.cc:729] T d4804397ef3448b8aac0ef154106fc96 P df6cca547d3847d08cf92cec310077c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6cca547d3847d08cf92cec310077c8, State: Initialized, Role: FOLLOWER
I20260928 22:49:27.215575 24897 consensus_queue.cc:260] T d4804397ef3448b8aac0ef154106fc96 P df6cca547d3847d08cf92cec310077c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } } peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } }
I20260928 22:49:27.215799 24902 tablet_bootstrap.cc:492] T d4804397ef3448b8aac0ef154106fc96 P 33643294a0eb433e8931f67583d665e8: Bootstrap starting.
I20260928 22:49:27.215934 24897 ts_tablet_manager.cc:1428] T d4804397ef3448b8aac0ef154106fc96 P df6cca547d3847d08cf92cec310077c8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:49:27.216086 24729 heartbeater.cc:503] Master 127.16.121.190:40321 was elected leader, sending a full tablet report...
I20260928 22:49:27.216837 24902 tablet_bootstrap.cc:654] T d4804397ef3448b8aac0ef154106fc96 P 33643294a0eb433e8931f67583d665e8: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:27.217643 24902 log.cc:824] T d4804397ef3448b8aac0ef154106fc96 P 33643294a0eb433e8931f67583d665e8: Log is configured to *not* fsync() on all Append() calls
I20260928 22:49:27.218649 24902 tablet_bootstrap.cc:492] T d4804397ef3448b8aac0ef154106fc96 P 33643294a0eb433e8931f67583d665e8: No bootstrap required, opened a new log
I20260928 22:49:27.218779 24902 ts_tablet_manager.cc:1397] T d4804397ef3448b8aac0ef154106fc96 P 33643294a0eb433e8931f67583d665e8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 22:49:27.220875 24902 raft_consensus.cc:348] T d4804397ef3448b8aac0ef154106fc96 P 33643294a0eb433e8931f67583d665e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } } peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } }
I20260928 22:49:27.221117 24902 raft_consensus.cc:374] T d4804397ef3448b8aac0ef154106fc96 P 33643294a0eb433e8931f67583d665e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:27.221167 24902 raft_consensus.cc:729] T d4804397ef3448b8aac0ef154106fc96 P 33643294a0eb433e8931f67583d665e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33643294a0eb433e8931f67583d665e8, State: Initialized, Role: FOLLOWER
I20260928 22:49:27.221308 24902 consensus_queue.cc:260] T d4804397ef3448b8aac0ef154106fc96 P 33643294a0eb433e8931f67583d665e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } } peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } }
I20260928 22:49:27.221639 24902 ts_tablet_manager.cc:1428] T d4804397ef3448b8aac0ef154106fc96 P 33643294a0eb433e8931f67583d665e8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:49:27.221714 24589 heartbeater.cc:503] Master 127.16.121.190:40321 was elected leader, sending a full tablet report...
I20260928 22:49:27.254868 24905 raft_consensus.cc:482] T d4804397ef3448b8aac0ef154106fc96 P 33643294a0eb433e8931f67583d665e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:49:27.255096 24905 raft_consensus.cc:504] T d4804397ef3448b8aac0ef154106fc96 P 33643294a0eb433e8931f67583d665e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } } peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } }
I20260928 22:49:27.255517 24905 leader_election.cc:288] T d4804397ef3448b8aac0ef154106fc96 P 33643294a0eb433e8931f67583d665e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82c52234a3a94312aae42cfb5ab3ff58 (127.16.121.131:36097), df6cca547d3847d08cf92cec310077c8 (127.16.121.130:38577)
I20260928 22:49:27.259701 24894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:27.255914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:33201 (local address 127.16.121.131:36097)
0928 22:49:27.255967 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:27.255969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:27.255984 (+ 15us) server_negotiation.cc:407] Connection header received
0928 22:49:27.256052 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:27.256053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:27.256064 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:27.256130 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:49:27.256364 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.257043 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.258344 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.259140 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.259324 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:27.259334 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:27.259552 (+ 218us) server_negotiation.cc:299] Negotiation successful
0928 22:49:27.259599 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 22:49:27.259739 24482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:27.255786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.131:36097 (local address 127.16.121.129:33201)
0928 22:49:27.255892 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:27.255904 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:27.255959 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:27.256149 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:27.256150 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:27.256152 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:49:27.256229 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:27.256230 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.257143 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.257144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:27.258232 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:27.258235 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.259251 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.259252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:27.259351 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:27.259356 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:49:27.259527 (+ 171us) client_negotiation.cc:240] Negotiation successful
0928 22:49:27.259601 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260928 22:49:27.259805 24896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:27.255934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:51091 (local address 127.16.121.130:38577)
0928 22:49:27.255978 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:27.255992 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:27.256117 (+ 125us) server_negotiation.cc:407] Connection header received
0928 22:49:27.256441 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:27.256444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:27.256457 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:27.256526 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:49:27.256767 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.257449 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.258722 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.259378 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.259445 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:27.259453 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:27.259673 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 22:49:27.259722 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 22:49:27.260108 24906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:27.255789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.130:38577 (local address 127.16.121.129:51091)
0928 22:49:27.256057 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:27.256069 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:27.256144 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:27.256549 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:27.256550 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:27.256553 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:49:27.256630 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:27.256632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.257556 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.257558 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:27.258612 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:27.258616 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.259467 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.259468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:27.259572 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:27.259576 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:49:27.259918 (+ 342us) client_negotiation.cc:240] Negotiation successful
0928 22:49:27.259994 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":24,"thread_start_us":148,"threads_started":1}
I20260928 22:49:27.260169 24823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4804397ef3448b8aac0ef154106fc96" candidate_uuid: "33643294a0eb433e8931f67583d665e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82c52234a3a94312aae42cfb5ab3ff58" is_pre_election: true
I20260928 22:49:27.260479 24823 raft_consensus.cc:2454] T d4804397ef3448b8aac0ef154106fc96 P 82c52234a3a94312aae42cfb5ab3ff58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33643294a0eb433e8931f67583d665e8 in term 0.
I20260928 22:49:27.260524 24683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4804397ef3448b8aac0ef154106fc96" candidate_uuid: "33643294a0eb433e8931f67583d665e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6cca547d3847d08cf92cec310077c8" is_pre_election: true
I20260928 22:49:27.260658 24683 raft_consensus.cc:2454] T d4804397ef3448b8aac0ef154106fc96 P df6cca547d3847d08cf92cec310077c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33643294a0eb433e8931f67583d665e8 in term 0.
I20260928 22:49:27.260859 24476 leader_election.cc:302] T d4804397ef3448b8aac0ef154106fc96 P 33643294a0eb433e8931f67583d665e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33643294a0eb433e8931f67583d665e8, 82c52234a3a94312aae42cfb5ab3ff58; no voters:
I20260928 22:49:27.261085 24905 raft_consensus.cc:2781] T d4804397ef3448b8aac0ef154106fc96 P 33643294a0eb433e8931f67583d665e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:49:27.261180 24905 raft_consensus.cc:482] T d4804397ef3448b8aac0ef154106fc96 P 33643294a0eb433e8931f67583d665e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:49:27.261217 24905 raft_consensus.cc:3037] T d4804397ef3448b8aac0ef154106fc96 P 33643294a0eb433e8931f67583d665e8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:27.262014 24905 raft_consensus.cc:504] T d4804397ef3448b8aac0ef154106fc96 P 33643294a0eb433e8931f67583d665e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } } peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } }
I20260928 22:49:27.262190 24905 leader_election.cc:288] T d4804397ef3448b8aac0ef154106fc96 P 33643294a0eb433e8931f67583d665e8 [CANDIDATE]: Term 1 election: Requested vote from peers 82c52234a3a94312aae42cfb5ab3ff58 (127.16.121.131:36097), df6cca547d3847d08cf92cec310077c8 (127.16.121.130:38577)
I20260928 22:49:27.262382 24683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4804397ef3448b8aac0ef154106fc96" candidate_uuid: "33643294a0eb433e8931f67583d665e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6cca547d3847d08cf92cec310077c8"
I20260928 22:49:27.262399 24823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4804397ef3448b8aac0ef154106fc96" candidate_uuid: "33643294a0eb433e8931f67583d665e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82c52234a3a94312aae42cfb5ab3ff58"
I20260928 22:49:27.262466 24683 raft_consensus.cc:3037] T d4804397ef3448b8aac0ef154106fc96 P df6cca547d3847d08cf92cec310077c8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:27.262472 24823 raft_consensus.cc:3037] T d4804397ef3448b8aac0ef154106fc96 P 82c52234a3a94312aae42cfb5ab3ff58 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:27.263405 24823 raft_consensus.cc:2454] T d4804397ef3448b8aac0ef154106fc96 P 82c52234a3a94312aae42cfb5ab3ff58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33643294a0eb433e8931f67583d665e8 in term 1.
I20260928 22:49:27.263405 24683 raft_consensus.cc:2454] T d4804397ef3448b8aac0ef154106fc96 P df6cca547d3847d08cf92cec310077c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33643294a0eb433e8931f67583d665e8 in term 1.
I20260928 22:49:27.263949 24474 leader_election.cc:302] T d4804397ef3448b8aac0ef154106fc96 P 33643294a0eb433e8931f67583d665e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33643294a0eb433e8931f67583d665e8, df6cca547d3847d08cf92cec310077c8; no voters:
I20260928 22:49:27.264087 24905 raft_consensus.cc:2781] T d4804397ef3448b8aac0ef154106fc96 P 33643294a0eb433e8931f67583d665e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:49:27.264236 24905 raft_consensus.cc:686] T d4804397ef3448b8aac0ef154106fc96 P 33643294a0eb433e8931f67583d665e8 [term 1 LEADER]: Becoming Leader. State: Replica: 33643294a0eb433e8931f67583d665e8, State: Running, Role: LEADER
I20260928 22:49:27.264344 24905 consensus_queue.cc:237] T d4804397ef3448b8aac0ef154106fc96 P 33643294a0eb433e8931f67583d665e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } } peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } }
I20260928 22:49:27.265230 24385 catalog_manager.cc:5495] T d4804397ef3448b8aac0ef154106fc96 P 33643294a0eb433e8931f67583d665e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33643294a0eb433e8931f67583d665e8 (127.16.121.129). New cstate: current_term: 1 leader_uuid: "33643294a0eb433e8931f67583d665e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } health_report { overall_health: UNKNOWN } } }
I20260928 22:49:27.281543 24895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:27.278291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52226 (local address 127.16.121.129:37109)
0928 22:49:27.278369 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:27.278371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:27.278381 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:49:27.278430 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:27.278431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:27.278449 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:27.278517 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:49:27.278725 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.279345 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.280412 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.280628 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.280702 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:27.280933 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:49:27.281065 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:49:27.281157 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:27.281372 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 22:49:27.281420 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260928 22:49:27.282979 24590 tablet.cc:2369] T d4804397ef3448b8aac0ef154106fc96 P 33643294a0eb433e8931f67583d665e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:49:27.284217 24823 raft_consensus.cc:1261] T d4804397ef3448b8aac0ef154106fc96 P 82c52234a3a94312aae42cfb5ab3ff58 [term 1 FOLLOWER]: Refusing update from remote peer 33643294a0eb433e8931f67583d665e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:49:27.284456 24683 raft_consensus.cc:1261] T d4804397ef3448b8aac0ef154106fc96 P df6cca547d3847d08cf92cec310077c8 [term 1 FOLLOWER]: Refusing update from remote peer 33643294a0eb433e8931f67583d665e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:49:27.284641 24907 consensus_queue.cc:1035] T d4804397ef3448b8aac0ef154106fc96 P 33643294a0eb433e8931f67583d665e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:49:27.284781 24905 consensus_queue.cc:1035] T d4804397ef3448b8aac0ef154106fc96 P 33643294a0eb433e8931f67583d665e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:49:27.287747 24910 mvcc.cc:205] Tried to move back new op lower bound from 7334444102792720384 to 7334444102715564032. Current Snapshot: MvccSnapshot[applied={T|T < 7334444102792720384}]
I20260928 22:49:27.287962 24912 mvcc.cc:205] Tried to move back new op lower bound from 7334444102792720384 to 7334444102715564032. Current Snapshot: MvccSnapshot[applied={T|T < 7334444102792720384}]
I20260928 22:49:27.291306 24385 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:49:27.291445 24385 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:49:27.292573 24385 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 20a5b16187d042b5ac6e0609f080bced: Sending DeleteTablet for 3 replicas of tablet d4804397ef3448b8aac0ef154106fc96
I20260928 22:49:27.293084 24523 tablet_service.cc:1514] Processing DeleteTablet for tablet d4804397ef3448b8aac0ef154106fc96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:49:27 UTC) from {username='oryx', principal='oryx/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:52210
I20260928 22:49:27.293152 24663 tablet_service.cc:1514] Processing DeleteTablet for tablet d4804397ef3448b8aac0ef154106fc96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:49:27 UTC) from {username='oryx', principal='oryx/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:53028
I20260928 22:49:27.293123 24803 tablet_service.cc:1514] Processing DeleteTablet for tablet d4804397ef3448b8aac0ef154106fc96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:49:27 UTC) from {username='oryx', principal='oryx/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:40098
I20260928 22:49:27.293429 24921 tablet_replica.cc:330] T d4804397ef3448b8aac0ef154106fc96 P df6cca547d3847d08cf92cec310077c8: stopping tablet replica
I20260928 22:49:27.293437 24920 tablet_replica.cc:330] T d4804397ef3448b8aac0ef154106fc96 P 33643294a0eb433e8931f67583d665e8: stopping tablet replica
I20260928 22:49:27.293561 24920 raft_consensus.cc:2229] T d4804397ef3448b8aac0ef154106fc96 P 33643294a0eb433e8931f67583d665e8 [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:49:27.293596 24921 raft_consensus.cc:2229] T d4804397ef3448b8aac0ef154106fc96 P df6cca547d3847d08cf92cec310077c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:49:27.293751 24920 raft_consensus.cc:2258] T d4804397ef3448b8aac0ef154106fc96 P 33643294a0eb433e8931f67583d665e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:49:27.293763 24921 raft_consensus.cc:2258] T d4804397ef3448b8aac0ef154106fc96 P df6cca547d3847d08cf92cec310077c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:49:27.293982 24922 tablet_replica.cc:330] T d4804397ef3448b8aac0ef154106fc96 P 82c52234a3a94312aae42cfb5ab3ff58: stopping tablet replica
I20260928 22:49:27.294145 24922 raft_consensus.cc:2229] T d4804397ef3448b8aac0ef154106fc96 P 82c52234a3a94312aae42cfb5ab3ff58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:49:27.294094 24385 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44112:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:49:27.294287 24385 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:49:27.294376 24920 ts_tablet_manager.cc:1905] T d4804397ef3448b8aac0ef154106fc96 P 33643294a0eb433e8931f67583d665e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:49:27.294593 24922 raft_consensus.cc:2258] T d4804397ef3448b8aac0ef154106fc96 P 82c52234a3a94312aae42cfb5ab3ff58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:49:27.295280 24921 ts_tablet_manager.cc:1905] T d4804397ef3448b8aac0ef154106fc96 P df6cca547d3847d08cf92cec310077c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:49:27.295606 24922 ts_tablet_manager.cc:1905] T d4804397ef3448b8aac0ef154106fc96 P 82c52234a3a94312aae42cfb5ab3ff58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:49:27.295764 24803 tablet_service.cc:1467] Processing CreateTablet for tablet 7d81d3e2a8fc47c49e7d8cd4378844ff (DEFAULT_TABLE table=test-table [id=81eb4c9fa9e843448691bb0add1f5539]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:49:27.295919 24803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d81d3e2a8fc47c49e7d8cd4378844ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:27.297719 24898 tablet_bootstrap.cc:492] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 82c52234a3a94312aae42cfb5ab3ff58: Bootstrap starting.
I20260928 22:49:27.298132 24898 tablet_bootstrap.cc:654] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 82c52234a3a94312aae42cfb5ab3ff58: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:27.298724 24922 ts_tablet_manager.cc:1918] T d4804397ef3448b8aac0ef154106fc96 P 82c52234a3a94312aae42cfb5ab3ff58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:49:27.298796 24921 ts_tablet_manager.cc:1918] T d4804397ef3448b8aac0ef154106fc96 P df6cca547d3847d08cf92cec310077c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:49:27.298883 24921 log.cc:1196] T d4804397ef3448b8aac0ef154106fc96 P df6cca547d3847d08cf92cec310077c8: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-1/wal/wals/d4804397ef3448b8aac0ef154106fc96
I20260928 22:49:27.299080 24523 tablet_service.cc:1467] Processing CreateTablet for tablet 7d81d3e2a8fc47c49e7d8cd4378844ff (DEFAULT_TABLE table=test-table [id=81eb4c9fa9e843448691bb0add1f5539]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:49:27.299105 24663 tablet_service.cc:1467] Processing CreateTablet for tablet 7d81d3e2a8fc47c49e7d8cd4378844ff (DEFAULT_TABLE table=test-table [id=81eb4c9fa9e843448691bb0add1f5539]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:49:27.299216 24523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d81d3e2a8fc47c49e7d8cd4378844ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:27.299270 24663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d81d3e2a8fc47c49e7d8cd4378844ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:27.301124 24922 log.cc:1196] T d4804397ef3448b8aac0ef154106fc96 P 82c52234a3a94312aae42cfb5ab3ff58: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-2/wal/wals/d4804397ef3448b8aac0ef154106fc96
I20260928 22:49:27.301406 24902 tablet_bootstrap.cc:492] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 33643294a0eb433e8931f67583d665e8: Bootstrap starting.
I20260928 22:49:27.301414 24897 tablet_bootstrap.cc:492] T 7d81d3e2a8fc47c49e7d8cd4378844ff P df6cca547d3847d08cf92cec310077c8: Bootstrap starting.
I20260928 22:49:27.301489 24922 ts_tablet_manager.cc:1939] T d4804397ef3448b8aac0ef154106fc96 P 82c52234a3a94312aae42cfb5ab3ff58: Deleting consensus metadata
I20260928 22:49:27.301833 24902 tablet_bootstrap.cc:654] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 33643294a0eb433e8931f67583d665e8: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:27.301889 24897 tablet_bootstrap.cc:654] T 7d81d3e2a8fc47c49e7d8cd4378844ff P df6cca547d3847d08cf92cec310077c8: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:27.302052 24372 catalog_manager.cc:4841] TS 82c52234a3a94312aae42cfb5ab3ff58 (127.16.121.131:36097): tablet d4804397ef3448b8aac0ef154106fc96 (table test-table [id=550e8b7a271046ac9aae6856f523b91b]) successfully deleted
I20260928 22:49:27.302388 24898 tablet_bootstrap.cc:492] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 82c52234a3a94312aae42cfb5ab3ff58: No bootstrap required, opened a new log
I20260928 22:49:27.302443 24898 ts_tablet_manager.cc:1397] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 82c52234a3a94312aae42cfb5ab3ff58: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260928 22:49:27.302562 24920 ts_tablet_manager.cc:1918] T d4804397ef3448b8aac0ef154106fc96 P 33643294a0eb433e8931f67583d665e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:49:27.302634 24920 log.cc:1196] T d4804397ef3448b8aac0ef154106fc96 P 33643294a0eb433e8931f67583d665e8: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-0/wal/wals/d4804397ef3448b8aac0ef154106fc96
I20260928 22:49:27.302613 24898 raft_consensus.cc:348] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 82c52234a3a94312aae42cfb5ab3ff58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } }
I20260928 22:49:27.302664 24921 ts_tablet_manager.cc:1939] T d4804397ef3448b8aac0ef154106fc96 P df6cca547d3847d08cf92cec310077c8: Deleting consensus metadata
I20260928 22:49:27.302692 24898 raft_consensus.cc:374] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 82c52234a3a94312aae42cfb5ab3ff58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:27.302716 24898 raft_consensus.cc:729] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 82c52234a3a94312aae42cfb5ab3ff58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82c52234a3a94312aae42cfb5ab3ff58, State: Initialized, Role: FOLLOWER
I20260928 22:49:27.302766 24898 consensus_queue.cc:260] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 82c52234a3a94312aae42cfb5ab3ff58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } }
I20260928 22:49:27.302870 24898 ts_tablet_manager.cc:1428] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 82c52234a3a94312aae42cfb5ab3ff58: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:27.303143 24370 catalog_manager.cc:4841] TS df6cca547d3847d08cf92cec310077c8 (127.16.121.130:38577): tablet d4804397ef3448b8aac0ef154106fc96 (table test-table [id=550e8b7a271046ac9aae6856f523b91b]) successfully deleted
I20260928 22:49:27.303645 24897 tablet_bootstrap.cc:492] T 7d81d3e2a8fc47c49e7d8cd4378844ff P df6cca547d3847d08cf92cec310077c8: No bootstrap required, opened a new log
I20260928 22:49:27.303702 24897 ts_tablet_manager.cc:1397] T 7d81d3e2a8fc47c49e7d8cd4378844ff P df6cca547d3847d08cf92cec310077c8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:49:27.303911 24920 ts_tablet_manager.cc:1939] T d4804397ef3448b8aac0ef154106fc96 P 33643294a0eb433e8931f67583d665e8: Deleting consensus metadata
I20260928 22:49:27.304054 24897 raft_consensus.cc:348] T 7d81d3e2a8fc47c49e7d8cd4378844ff P df6cca547d3847d08cf92cec310077c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } }
I20260928 22:49:27.304137 24897 raft_consensus.cc:374] T 7d81d3e2a8fc47c49e7d8cd4378844ff P df6cca547d3847d08cf92cec310077c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:27.304159 24897 raft_consensus.cc:729] T 7d81d3e2a8fc47c49e7d8cd4378844ff P df6cca547d3847d08cf92cec310077c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6cca547d3847d08cf92cec310077c8, State: Initialized, Role: FOLLOWER
I20260928 22:49:27.304208 24897 consensus_queue.cc:260] T 7d81d3e2a8fc47c49e7d8cd4378844ff P df6cca547d3847d08cf92cec310077c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } }
I20260928 22:49:27.304291 24897 ts_tablet_manager.cc:1428] T 7d81d3e2a8fc47c49e7d8cd4378844ff P df6cca547d3847d08cf92cec310077c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:27.304270 24370 catalog_manager.cc:4841] TS 33643294a0eb433e8931f67583d665e8 (127.16.121.129:37109): tablet d4804397ef3448b8aac0ef154106fc96 (table test-table [id=550e8b7a271046ac9aae6856f523b91b]) successfully deleted
I20260928 22:49:27.308977 24902 tablet_bootstrap.cc:492] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 33643294a0eb433e8931f67583d665e8: No bootstrap required, opened a new log
I20260928 22:49:27.309041 24902 ts_tablet_manager.cc:1397] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 33643294a0eb433e8931f67583d665e8: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260928 22:49:27.309219 24902 raft_consensus.cc:348] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 33643294a0eb433e8931f67583d665e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } }
I20260928 22:49:27.309356 24902 raft_consensus.cc:374] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 33643294a0eb433e8931f67583d665e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:27.309381 24902 raft_consensus.cc:729] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 33643294a0eb433e8931f67583d665e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33643294a0eb433e8931f67583d665e8, State: Initialized, Role: FOLLOWER
I20260928 22:49:27.309439 24902 consensus_queue.cc:260] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 33643294a0eb433e8931f67583d665e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } }
I20260928 22:49:27.309563 24902 ts_tablet_manager.cc:1428] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 33643294a0eb433e8931f67583d665e8: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 22:49:27.339568 24901 raft_consensus.cc:482] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 82c52234a3a94312aae42cfb5ab3ff58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:49:27.339673 24901 raft_consensus.cc:504] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 82c52234a3a94312aae42cfb5ab3ff58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } }
I20260928 22:49:27.340148 24901 leader_election.cc:288] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 82c52234a3a94312aae42cfb5ab3ff58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33643294a0eb433e8931f67583d665e8 (127.16.121.129:37109), df6cca547d3847d08cf92cec310077c8 (127.16.121.130:38577)
I20260928 22:49:27.342006 24903 raft_consensus.cc:482] T 7d81d3e2a8fc47c49e7d8cd4378844ff P df6cca547d3847d08cf92cec310077c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:49:27.342088 24903 raft_consensus.cc:504] T 7d81d3e2a8fc47c49e7d8cd4378844ff P df6cca547d3847d08cf92cec310077c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } }
I20260928 22:49:27.342473 24903 leader_election.cc:288] T 7d81d3e2a8fc47c49e7d8cd4378844ff P df6cca547d3847d08cf92cec310077c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33643294a0eb433e8931f67583d665e8 (127.16.121.129:37109), 82c52234a3a94312aae42cfb5ab3ff58 (127.16.121.131:36097)
I20260928 22:49:27.344530 24762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:27.340341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:37109 (local address 127.16.121.131:42155)
0928 22:49:27.340420 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:27.340434 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:27.340489 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:27.340848 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:27.340849 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:27.340851 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:49:27.340927 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:27.340929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.341771 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.341772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:27.343226 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:27.343230 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.344238 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.344240 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:27.344330 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:27.344335 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:49:27.344391 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 22:49:27.344441 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 22:49:27.344772 24896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:27.340665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:56521 (local address 127.16.121.130:38577)
0928 22:49:27.340718 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:27.340720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:27.340785 (+ 65us) server_negotiation.cc:407] Connection header received
0928 22:49:27.340907 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:27.340908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:27.340918 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:27.341001 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:49:27.341188 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.342010 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.343682 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.344308 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.344362 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:27.344370 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:27.344652 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 22:49:27.344697 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 22:49:27.345060 24895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:27.340601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:42155 (local address 127.16.121.129:37109)
0928 22:49:27.340652 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:27.340655 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:27.340664 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:49:27.340697 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:27.340698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:27.340709 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:27.340796 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:49:27.341074 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.341672 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.343417 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.344122 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.344194 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:27.344204 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:27.344905 (+ 701us) server_negotiation.cc:299] Negotiation successful
0928 22:49:27.344976 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 22:49:27.345304 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d81d3e2a8fc47c49e7d8cd4378844ff" candidate_uuid: "82c52234a3a94312aae42cfb5ab3ff58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33643294a0eb433e8931f67583d665e8" is_pre_election: true
I20260928 22:49:27.345376 24926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:27.340581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.130:38577 (local address 127.16.121.131:56521)
0928 22:49:27.340713 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:27.340724 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:27.340821 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:27.341016 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:27.341017 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:27.341019 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:49:27.341094 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:27.341096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.342127 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.342128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:27.343563 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:27.343566 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.344477 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.344479 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:27.344558 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:27.344562 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:49:27.345177 (+ 615us) client_negotiation.cc:240] Negotiation successful
0928 22:49:27.345246 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":72,"threads_started":1}
I20260928 22:49:27.345435 24543 raft_consensus.cc:2454] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 33643294a0eb433e8931f67583d665e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82c52234a3a94312aae42cfb5ab3ff58 in term 0.
I20260928 22:49:27.345633 24683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d81d3e2a8fc47c49e7d8cd4378844ff" candidate_uuid: "82c52234a3a94312aae42cfb5ab3ff58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6cca547d3847d08cf92cec310077c8" is_pre_election: true
I20260928 22:49:27.345657 24754 leader_election.cc:302] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 82c52234a3a94312aae42cfb5ab3ff58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33643294a0eb433e8931f67583d665e8, 82c52234a3a94312aae42cfb5ab3ff58; no voters:
I20260928 22:49:27.345714 24683 raft_consensus.cc:2454] T 7d81d3e2a8fc47c49e7d8cd4378844ff P df6cca547d3847d08cf92cec310077c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82c52234a3a94312aae42cfb5ab3ff58 in term 0.
I20260928 22:49:27.345788 24901 raft_consensus.cc:2781] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 82c52234a3a94312aae42cfb5ab3ff58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:49:27.345815 24901 raft_consensus.cc:482] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 82c52234a3a94312aae42cfb5ab3ff58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:49:27.345836 24901 raft_consensus.cc:3037] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 82c52234a3a94312aae42cfb5ab3ff58 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:27.346813 24901 raft_consensus.cc:504] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 82c52234a3a94312aae42cfb5ab3ff58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } }
I20260928 22:49:27.346949 24901 leader_election.cc:288] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 82c52234a3a94312aae42cfb5ab3ff58 [CANDIDATE]: Term 1 election: Requested vote from peers 33643294a0eb433e8931f67583d665e8 (127.16.121.129:37109), df6cca547d3847d08cf92cec310077c8 (127.16.121.130:38577)
I20260928 22:49:27.347527 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d81d3e2a8fc47c49e7d8cd4378844ff" candidate_uuid: "82c52234a3a94312aae42cfb5ab3ff58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33643294a0eb433e8931f67583d665e8"
I20260928 22:49:27.347616 24543 raft_consensus.cc:3037] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 33643294a0eb433e8931f67583d665e8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:27.347636 24622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:27.342682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:37109 (local address 127.16.121.130:42741)
0928 22:49:27.342731 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:27.342742 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:27.342781 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:27.343830 (+ 1049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:27.343831 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:27.343834 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:49:27.343899 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:27.343901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.344987 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.344988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:27.346361 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:27.346364 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.347342 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.347344 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:27.347440 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:27.347444 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:49:27.347507 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 22:49:27.347553 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 22:49:27.347669 24683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d81d3e2a8fc47c49e7d8cd4378844ff" candidate_uuid: "82c52234a3a94312aae42cfb5ab3ff58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6cca547d3847d08cf92cec310077c8"
I20260928 22:49:27.347748 24683 raft_consensus.cc:3037] T 7d81d3e2a8fc47c49e7d8cd4378844ff P df6cca547d3847d08cf92cec310077c8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:27.348546 24543 raft_consensus.cc:2454] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 33643294a0eb433e8931f67583d665e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82c52234a3a94312aae42cfb5ab3ff58 in term 1.
I20260928 22:49:27.348553 24683 raft_consensus.cc:2454] T 7d81d3e2a8fc47c49e7d8cd4378844ff P df6cca547d3847d08cf92cec310077c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82c52234a3a94312aae42cfb5ab3ff58 in term 1.
I20260928 22:49:27.348872 24754 leader_election.cc:302] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 82c52234a3a94312aae42cfb5ab3ff58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82c52234a3a94312aae42cfb5ab3ff58, df6cca547d3847d08cf92cec310077c8; no voters:
I20260928 22:49:27.349015 24901 raft_consensus.cc:2781] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 82c52234a3a94312aae42cfb5ab3ff58 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:49:27.349176 24901 raft_consensus.cc:686] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 82c52234a3a94312aae42cfb5ab3ff58 [term 1 LEADER]: Becoming Leader. State: Replica: 82c52234a3a94312aae42cfb5ab3ff58, State: Running, Role: LEADER
I20260928 22:49:27.349243 24927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:27.342918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:42741 (local address 127.16.121.129:37109)
0928 22:49:27.343671 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:27.343673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:27.343683 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:49:27.343707 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:27.343708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:27.343718 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:27.343973 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:49:27.344023 (+ 50us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.344881 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.346494 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.347227 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.347337 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:27.347346 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:27.349100 (+ 1754us) server_negotiation.cc:299] Negotiation successful
0928 22:49:27.349158 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":733,"thread_start_us":84,"threads_started":1}
I20260928 22:49:27.349273 24901 consensus_queue.cc:237] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 82c52234a3a94312aae42cfb5ab3ff58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } }
I20260928 22:49:27.349380 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d81d3e2a8fc47c49e7d8cd4378844ff" candidate_uuid: "df6cca547d3847d08cf92cec310077c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33643294a0eb433e8931f67583d665e8" is_pre_election: true
I20260928 22:49:27.349478 24543 raft_consensus.cc:2379] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 33643294a0eb433e8931f67583d665e8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6cca547d3847d08cf92cec310077c8 in current term 1: Already voted for candidate 82c52234a3a94312aae42cfb5ab3ff58 in this term.
I20260928 22:49:27.350124 24385 catalog_manager.cc:5495] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 82c52234a3a94312aae42cfb5ab3ff58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82c52234a3a94312aae42cfb5ab3ff58 (127.16.121.131). New cstate: current_term: 1 leader_uuid: "82c52234a3a94312aae42cfb5ab3ff58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } health_report { overall_health: HEALTHY } } }
I20260928 22:49:27.351174 24622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:27.343162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.131:36097 (local address 127.16.121.130:46013)
0928 22:49:27.347706 (+ 4544us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:27.347718 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:27.347763 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:27.347958 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:27.347959 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:27.347960 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:49:27.347998 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:27.348000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.348912 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.348914 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:27.350037 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:27.350040 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.350836 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.350837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:27.350925 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:27.350929 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:49:27.351029 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 22:49:27.351094 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4532,"thread_start_us":60,"threads_started":1}
I20260928 22:49:27.351246 24894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:27.343309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:46013 (local address 127.16.121.131:36097)
0928 22:49:27.343469 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:27.343471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:27.347822 (+ 4351us) server_negotiation.cc:407] Connection header received
0928 22:49:27.347848 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:27.347849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:27.347859 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:27.347920 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:49:27.348069 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.348802 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.350142 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.350731 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.350798 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:27.350807 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:27.351038 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 22:49:27.351175 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260928 22:49:27.351403 24823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d81d3e2a8fc47c49e7d8cd4378844ff" candidate_uuid: "df6cca547d3847d08cf92cec310077c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82c52234a3a94312aae42cfb5ab3ff58" is_pre_election: true
I20260928 22:49:27.351676 24616 leader_election.cc:302] T 7d81d3e2a8fc47c49e7d8cd4378844ff P df6cca547d3847d08cf92cec310077c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6cca547d3847d08cf92cec310077c8; no voters: 33643294a0eb433e8931f67583d665e8, 82c52234a3a94312aae42cfb5ab3ff58
I20260928 22:49:27.351821 24903 raft_consensus.cc:2726] T 7d81d3e2a8fc47c49e7d8cd4378844ff P df6cca547d3847d08cf92cec310077c8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
Sep 28 22:49:27 dist-test-slave-56vw krb5kdc[24341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635766, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.121.190@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM
I20260928 22:49:27.392812 24936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:27.383977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:40321 (local address 127.0.0.1:44126)
0928 22:49:27.384182 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:27.384192 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:27.384255 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:27.384448 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:27.384449 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:27.384789 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:27.384847 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:27.384850 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.385818 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.385820 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:27.386432 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:27.386435 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.386989 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.386990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:27.387154 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:27.388159 (+ 1005us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:27.388165 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:27.389731 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:27.391602 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:27.391610 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:27.391612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:27.391823 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:27.392073 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:27.392073 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:27.392074 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:27.392133 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:27.392458 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:27.392466 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:27.392541 (+ 75us) client_negotiation.cc:769] Sending connection context
0928 22:49:27.392610 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 22:49:27.392723 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":113,"threads_started":1}
I20260928 22:49:27.392918 24453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:27.384079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44126 (local address 127.16.121.190:40321)
0928 22:49:27.384149 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:27.384151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:27.384258 (+ 107us) server_negotiation.cc:407] Connection header received
0928 22:49:27.384313 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:27.384314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:27.384327 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:27.384412 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:27.384955 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.385701 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.386537 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.386863 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.386959 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:27.389843 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:27.389845 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:27.389845 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:27.389855 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:27.391510 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:27.391919 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:27.391920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:27.391921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:27.391988 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:27.392205 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:27.392206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:27.392207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:27.392358 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:27.392437 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:27.392667 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 22:49:27.392793 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":41}
W20260928 22:49:27.395625 24870 tablet.cc:2369] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 82c52234a3a94312aae42cfb5ab3ff58: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:49:27.395567 24385 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:44126:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 22:49:27.395813 24385 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:49:27.397567 24663 tablet_service.cc:1467] Processing CreateTablet for tablet 21e8ee5e013144bf85441abc6b2691af (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3ebdcdd121c9496d9e1870467a36b583]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 22:49:27.397614 24803 tablet_service.cc:1467] Processing CreateTablet for tablet 21e8ee5e013144bf85441abc6b2691af (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3ebdcdd121c9496d9e1870467a36b583]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 22:49:27.397624 24523 tablet_service.cc:1467] Processing CreateTablet for tablet 21e8ee5e013144bf85441abc6b2691af (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3ebdcdd121c9496d9e1870467a36b583]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 22:49:27.397763 24803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21e8ee5e013144bf85441abc6b2691af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:27.397763 24663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21e8ee5e013144bf85441abc6b2691af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:27.397773 24523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21e8ee5e013144bf85441abc6b2691af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:27.399446 24902 tablet_bootstrap.cc:492] T 21e8ee5e013144bf85441abc6b2691af P 33643294a0eb433e8931f67583d665e8: Bootstrap starting.
I20260928 22:49:27.400210 24902 tablet_bootstrap.cc:654] T 21e8ee5e013144bf85441abc6b2691af P 33643294a0eb433e8931f67583d665e8: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:27.400594 24898 tablet_bootstrap.cc:492] T 21e8ee5e013144bf85441abc6b2691af P 82c52234a3a94312aae42cfb5ab3ff58: Bootstrap starting.
I20260928 22:49:27.401021 24902 tablet_bootstrap.cc:492] T 21e8ee5e013144bf85441abc6b2691af P 33643294a0eb433e8931f67583d665e8: No bootstrap required, opened a new log
I20260928 22:49:27.401101 24902 ts_tablet_manager.cc:1397] T 21e8ee5e013144bf85441abc6b2691af P 33643294a0eb433e8931f67583d665e8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:49:27.401110 24898 tablet_bootstrap.cc:654] T 21e8ee5e013144bf85441abc6b2691af P 82c52234a3a94312aae42cfb5ab3ff58: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:27.401278 24902 raft_consensus.cc:348] T 21e8ee5e013144bf85441abc6b2691af P 33643294a0eb433e8931f67583d665e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } }
I20260928 22:49:27.401343 24902 raft_consensus.cc:374] T 21e8ee5e013144bf85441abc6b2691af P 33643294a0eb433e8931f67583d665e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:27.401360 24902 raft_consensus.cc:729] T 21e8ee5e013144bf85441abc6b2691af P 33643294a0eb433e8931f67583d665e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33643294a0eb433e8931f67583d665e8, State: Initialized, Role: FOLLOWER
I20260928 22:49:27.401419 24902 consensus_queue.cc:260] T 21e8ee5e013144bf85441abc6b2691af P 33643294a0eb433e8931f67583d665e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } }
I20260928 22:49:27.401500 24897 tablet_bootstrap.cc:492] T 21e8ee5e013144bf85441abc6b2691af P df6cca547d3847d08cf92cec310077c8: Bootstrap starting.
I20260928 22:49:27.401542 24902 ts_tablet_manager.cc:1428] T 21e8ee5e013144bf85441abc6b2691af P 33643294a0eb433e8931f67583d665e8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:27.401582 24915 tablet_replica.cc:439] T 21e8ee5e013144bf85441abc6b2691af P 33643294a0eb433e8931f67583d665e8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } } }
I20260928 22:49:27.401674 24915 tablet_replica.cc:442] T 21e8ee5e013144bf85441abc6b2691af P 33643294a0eb433e8931f67583d665e8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:49:27.401933 24897 tablet_bootstrap.cc:654] T 21e8ee5e013144bf85441abc6b2691af P df6cca547d3847d08cf92cec310077c8: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:27.402715 24897 tablet_bootstrap.cc:492] T 21e8ee5e013144bf85441abc6b2691af P df6cca547d3847d08cf92cec310077c8: No bootstrap required, opened a new log
I20260928 22:49:27.402787 24897 ts_tablet_manager.cc:1397] T 21e8ee5e013144bf85441abc6b2691af P df6cca547d3847d08cf92cec310077c8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:27.402961 24897 raft_consensus.cc:348] T 21e8ee5e013144bf85441abc6b2691af P df6cca547d3847d08cf92cec310077c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } }
I20260928 22:49:27.403077 24897 raft_consensus.cc:374] T 21e8ee5e013144bf85441abc6b2691af P df6cca547d3847d08cf92cec310077c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:27.403102 24897 raft_consensus.cc:729] T 21e8ee5e013144bf85441abc6b2691af P df6cca547d3847d08cf92cec310077c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6cca547d3847d08cf92cec310077c8, State: Initialized, Role: FOLLOWER
I20260928 22:49:27.403151 24897 consensus_queue.cc:260] T 21e8ee5e013144bf85441abc6b2691af P df6cca547d3847d08cf92cec310077c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } }
I20260928 22:49:27.403257 24897 ts_tablet_manager.cc:1428] T 21e8ee5e013144bf85441abc6b2691af P df6cca547d3847d08cf92cec310077c8: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 22:49:27.403331 24903 tablet_replica.cc:439] T 21e8ee5e013144bf85441abc6b2691af P df6cca547d3847d08cf92cec310077c8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } } }
I20260928 22:49:27.403544 24903 tablet_replica.cc:442] T 21e8ee5e013144bf85441abc6b2691af P df6cca547d3847d08cf92cec310077c8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:49:27.403692 24898 tablet_bootstrap.cc:492] T 21e8ee5e013144bf85441abc6b2691af P 82c52234a3a94312aae42cfb5ab3ff58: No bootstrap required, opened a new log
I20260928 22:49:27.403764 24898 ts_tablet_manager.cc:1397] T 21e8ee5e013144bf85441abc6b2691af P 82c52234a3a94312aae42cfb5ab3ff58: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260928 22:49:27.404116 24898 raft_consensus.cc:348] T 21e8ee5e013144bf85441abc6b2691af P 82c52234a3a94312aae42cfb5ab3ff58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } }
I20260928 22:49:27.404201 24898 raft_consensus.cc:374] T 21e8ee5e013144bf85441abc6b2691af P 82c52234a3a94312aae42cfb5ab3ff58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:27.404222 24898 raft_consensus.cc:729] T 21e8ee5e013144bf85441abc6b2691af P 82c52234a3a94312aae42cfb5ab3ff58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82c52234a3a94312aae42cfb5ab3ff58, State: Initialized, Role: FOLLOWER
I20260928 22:49:27.404264 24898 consensus_queue.cc:260] T 21e8ee5e013144bf85441abc6b2691af P 82c52234a3a94312aae42cfb5ab3ff58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } }
I20260928 22:49:27.404383 24898 ts_tablet_manager.cc:1428] T 21e8ee5e013144bf85441abc6b2691af P 82c52234a3a94312aae42cfb5ab3ff58: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 22:49:27.404448 24929 tablet_replica.cc:439] T 21e8ee5e013144bf85441abc6b2691af P 82c52234a3a94312aae42cfb5ab3ff58: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } } }
I20260928 22:49:27.404542 24929 tablet_replica.cc:442] T 21e8ee5e013144bf85441abc6b2691af P 82c52234a3a94312aae42cfb5ab3ff58: This TxnStatusTablet replica's current role is: FOLLOWER
W20260928 22:49:27.457065 24730 tablet.cc:2369] T 21e8ee5e013144bf85441abc6b2691af P df6cca547d3847d08cf92cec310077c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:49:27.766780 24929 raft_consensus.cc:482] T 21e8ee5e013144bf85441abc6b2691af P 82c52234a3a94312aae42cfb5ab3ff58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:49:27.766929 24929 raft_consensus.cc:504] T 21e8ee5e013144bf85441abc6b2691af P 82c52234a3a94312aae42cfb5ab3ff58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } }
I20260928 22:49:27.767155 24929 leader_election.cc:288] T 21e8ee5e013144bf85441abc6b2691af P 82c52234a3a94312aae42cfb5ab3ff58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33643294a0eb433e8931f67583d665e8 (127.16.121.129:37109), df6cca547d3847d08cf92cec310077c8 (127.16.121.130:38577)
I20260928 22:49:27.767616 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e8ee5e013144bf85441abc6b2691af" candidate_uuid: "82c52234a3a94312aae42cfb5ab3ff58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33643294a0eb433e8931f67583d665e8" is_pre_election: true
I20260928 22:49:27.767688 24683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e8ee5e013144bf85441abc6b2691af" candidate_uuid: "82c52234a3a94312aae42cfb5ab3ff58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6cca547d3847d08cf92cec310077c8" is_pre_election: true
I20260928 22:49:27.767799 24683 raft_consensus.cc:2454] T 21e8ee5e013144bf85441abc6b2691af P df6cca547d3847d08cf92cec310077c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82c52234a3a94312aae42cfb5ab3ff58 in term 0.
I20260928 22:49:27.767745 24543 raft_consensus.cc:2454] T 21e8ee5e013144bf85441abc6b2691af P 33643294a0eb433e8931f67583d665e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82c52234a3a94312aae42cfb5ab3ff58 in term 0.
I20260928 22:49:27.767961 24754 leader_election.cc:302] T 21e8ee5e013144bf85441abc6b2691af P 82c52234a3a94312aae42cfb5ab3ff58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82c52234a3a94312aae42cfb5ab3ff58, df6cca547d3847d08cf92cec310077c8; no voters:
I20260928 22:49:27.768064 24929 raft_consensus.cc:2781] T 21e8ee5e013144bf85441abc6b2691af P 82c52234a3a94312aae42cfb5ab3ff58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:49:27.768139 24929 raft_consensus.cc:482] T 21e8ee5e013144bf85441abc6b2691af P 82c52234a3a94312aae42cfb5ab3ff58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:49:27.768167 24929 raft_consensus.cc:3037] T 21e8ee5e013144bf85441abc6b2691af P 82c52234a3a94312aae42cfb5ab3ff58 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:27.769307 24929 raft_consensus.cc:504] T 21e8ee5e013144bf85441abc6b2691af P 82c52234a3a94312aae42cfb5ab3ff58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } }
I20260928 22:49:27.769472 24929 leader_election.cc:288] T 21e8ee5e013144bf85441abc6b2691af P 82c52234a3a94312aae42cfb5ab3ff58 [CANDIDATE]: Term 1 election: Requested vote from peers 33643294a0eb433e8931f67583d665e8 (127.16.121.129:37109), df6cca547d3847d08cf92cec310077c8 (127.16.121.130:38577)
I20260928 22:49:27.769721 24683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e8ee5e013144bf85441abc6b2691af" candidate_uuid: "82c52234a3a94312aae42cfb5ab3ff58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6cca547d3847d08cf92cec310077c8"
I20260928 22:49:27.769809 24683 raft_consensus.cc:3037] T 21e8ee5e013144bf85441abc6b2691af P df6cca547d3847d08cf92cec310077c8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:27.769789 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e8ee5e013144bf85441abc6b2691af" candidate_uuid: "82c52234a3a94312aae42cfb5ab3ff58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33643294a0eb433e8931f67583d665e8"
I20260928 22:49:27.769889 24543 raft_consensus.cc:3037] T 21e8ee5e013144bf85441abc6b2691af P 33643294a0eb433e8931f67583d665e8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:27.770606 24683 raft_consensus.cc:2454] T 21e8ee5e013144bf85441abc6b2691af P df6cca547d3847d08cf92cec310077c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82c52234a3a94312aae42cfb5ab3ff58 in term 1.
I20260928 22:49:27.770787 24543 raft_consensus.cc:2454] T 21e8ee5e013144bf85441abc6b2691af P 33643294a0eb433e8931f67583d665e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82c52234a3a94312aae42cfb5ab3ff58 in term 1.
I20260928 22:49:27.770807 24754 leader_election.cc:302] T 21e8ee5e013144bf85441abc6b2691af P 82c52234a3a94312aae42cfb5ab3ff58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82c52234a3a94312aae42cfb5ab3ff58, df6cca547d3847d08cf92cec310077c8; no voters:
I20260928 22:49:27.770889 24929 raft_consensus.cc:2781] T 21e8ee5e013144bf85441abc6b2691af P 82c52234a3a94312aae42cfb5ab3ff58 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:49:27.770931 24929 raft_consensus.cc:686] T 21e8ee5e013144bf85441abc6b2691af P 82c52234a3a94312aae42cfb5ab3ff58 [term 1 LEADER]: Becoming Leader. State: Replica: 82c52234a3a94312aae42cfb5ab3ff58, State: Running, Role: LEADER
I20260928 22:49:27.770977 24929 consensus_queue.cc:237] T 21e8ee5e013144bf85441abc6b2691af P 82c52234a3a94312aae42cfb5ab3ff58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } }
I20260928 22:49:27.771274 24901 tablet_replica.cc:439] T 21e8ee5e013144bf85441abc6b2691af P 82c52234a3a94312aae42cfb5ab3ff58: TxnStatusTablet state changed. Reason: New leader 82c52234a3a94312aae42cfb5ab3ff58. Latest consensus state: current_term: 1 leader_uuid: "82c52234a3a94312aae42cfb5ab3ff58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } } }
I20260928 22:49:27.771348 24901 tablet_replica.cc:442] T 21e8ee5e013144bf85441abc6b2691af P 82c52234a3a94312aae42cfb5ab3ff58: This TxnStatusTablet replica's current role is: LEADER
I20260928 22:49:27.771492 24941 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 22:49:27.771598 24941 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 22:49:27.771730 24386 catalog_manager.cc:5495] T 21e8ee5e013144bf85441abc6b2691af P 82c52234a3a94312aae42cfb5ab3ff58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82c52234a3a94312aae42cfb5ab3ff58 (127.16.121.131). New cstate: current_term: 1 leader_uuid: "82c52234a3a94312aae42cfb5ab3ff58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } health_report { overall_health: HEALTHY } } }
I20260928 22:49:27.849676 24936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:27.845972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.131:36097 (local address 127.0.0.1:40100)
0928 22:49:27.846064 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:27.846076 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:27.846127 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:27.846427 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:27.846428 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:27.846430 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 22:49:27.846488 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:27.846491 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.847365 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.847367 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:27.848526 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:27.848530 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.849040 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.849041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:27.849118 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:27.849142 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 22:49:27.849463 (+ 321us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 22:49:27.849477 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 22:49:27.849534 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 22:49:27.849596 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 22:49:27.849653 24894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:27.846048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40100 (local address 127.16.121.131:36097)
0928 22:49:27.846103 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:27.846105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:27.846117 (+ 12us) server_negotiation.cc:407] Connection header received
0928 22:49:27.846214 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:27.846215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:27.846233 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:27.846394 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:49:27.846591 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.847246 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.848632 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.848940 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.849037 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:27.849251 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:49:27.849380 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:49:27.849483 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:27.849537 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 22:49:27.849560 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 22:49:27.851344 24683 raft_consensus.cc:1261] T 21e8ee5e013144bf85441abc6b2691af P df6cca547d3847d08cf92cec310077c8 [term 1 FOLLOWER]: Refusing update from remote peer 82c52234a3a94312aae42cfb5ab3ff58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:49:27.851452 24543 raft_consensus.cc:1261] T 21e8ee5e013144bf85441abc6b2691af P 33643294a0eb433e8931f67583d665e8 [term 1 FOLLOWER]: Refusing update from remote peer 82c52234a3a94312aae42cfb5ab3ff58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:49:27.851904 24929 consensus_queue.cc:1035] T 21e8ee5e013144bf85441abc6b2691af P 82c52234a3a94312aae42cfb5ab3ff58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:49:27.852100 24929 consensus_queue.cc:1035] T 21e8ee5e013144bf85441abc6b2691af P 82c52234a3a94312aae42cfb5ab3ff58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:49:27.852746 24915 tablet_replica.cc:439] T 21e8ee5e013144bf85441abc6b2691af P 33643294a0eb433e8931f67583d665e8: TxnStatusTablet state changed. Reason: New leader 82c52234a3a94312aae42cfb5ab3ff58. Latest consensus state: current_term: 1 leader_uuid: "82c52234a3a94312aae42cfb5ab3ff58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } } }
I20260928 22:49:27.852864 24915 tablet_replica.cc:442] T 21e8ee5e013144bf85441abc6b2691af P 33643294a0eb433e8931f67583d665e8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:49:27.853354 24903 tablet_replica.cc:439] T 21e8ee5e013144bf85441abc6b2691af P df6cca547d3847d08cf92cec310077c8: TxnStatusTablet state changed. Reason: New leader 82c52234a3a94312aae42cfb5ab3ff58. Latest consensus state: current_term: 1 leader_uuid: "82c52234a3a94312aae42cfb5ab3ff58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } } }
I20260928 22:49:27.853448 24903 tablet_replica.cc:442] T 21e8ee5e013144bf85441abc6b2691af P df6cca547d3847d08cf92cec310077c8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:49:27.853910 24901 tablet_replica.cc:439] T 21e8ee5e013144bf85441abc6b2691af P 82c52234a3a94312aae42cfb5ab3ff58: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "82c52234a3a94312aae42cfb5ab3ff58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } } }
I20260928 22:49:27.854014 24901 tablet_replica.cc:442] T 21e8ee5e013144bf85441abc6b2691af P 82c52234a3a94312aae42cfb5ab3ff58: This TxnStatusTablet replica's current role is: LEADER
I20260928 22:49:27.854571 24915 tablet_replica.cc:439] T 21e8ee5e013144bf85441abc6b2691af P 33643294a0eb433e8931f67583d665e8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "82c52234a3a94312aae42cfb5ab3ff58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } } }
I20260928 22:49:27.854662 24915 tablet_replica.cc:442] T 21e8ee5e013144bf85441abc6b2691af P 33643294a0eb433e8931f67583d665e8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:49:27.855211 24929 tablet_replica.cc:439] T 21e8ee5e013144bf85441abc6b2691af P 82c52234a3a94312aae42cfb5ab3ff58: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "82c52234a3a94312aae42cfb5ab3ff58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } } }
I20260928 22:49:27.855304 24929 tablet_replica.cc:442] T 21e8ee5e013144bf85441abc6b2691af P 82c52234a3a94312aae42cfb5ab3ff58: This TxnStatusTablet replica's current role is: LEADER
I20260928 22:49:27.855759 24903 tablet_replica.cc:439] T 21e8ee5e013144bf85441abc6b2691af P df6cca547d3847d08cf92cec310077c8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "82c52234a3a94312aae42cfb5ab3ff58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } } }
I20260928 22:49:27.855867 24903 tablet_replica.cc:442] T 21e8ee5e013144bf85441abc6b2691af P df6cca547d3847d08cf92cec310077c8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 22:49:27.859895 24894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:27.857059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40112 (local address 127.16.121.131:36097)
0928 22:49:27.857103 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:27.857106 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:27.857226 (+ 120us) server_negotiation.cc:407] Connection header received
0928 22:49:27.857305 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:27.857306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:27.857314 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:27.857409 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:49:27.857608 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.858135 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.859121 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.859341 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.859443 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:27.859612 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:49:27.859653 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:49:27.859721 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:27.859779 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 22:49:27.859802 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:49:27.867239 24894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:27.863836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:39159 (local address 127.16.121.131:36097)
0928 22:49:27.863882 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:27.863885 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:27.864135 (+ 250us) server_negotiation.cc:407] Connection header received
0928 22:49:27.864248 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:27.864249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:27.864262 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:27.864366 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:49:27.864533 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.865203 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.866394 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.866601 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.866674 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:27.866859 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:49:27.866889 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:49:27.866937 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:27.867014 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 22:49:27.867035 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":26}
I20260928 22:49:27.867607 24956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:27.863835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.131:36097 (local address 127.16.121.131:39159)
0928 22:49:27.864052 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:27.864064 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:27.864161 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:27.864373 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:27.864374 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:27.864375 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 22:49:27.864451 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:27.864453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.865301 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.865302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:27.866192 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:27.866194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.866690 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.866691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:27.866751 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:27.866768 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 22:49:27.866954 (+ 186us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 22:49:27.866956 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 22:49:27.867011 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 22:49:27.867101 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":131,"threads_started":1}
I20260928 22:49:27.870076 24543 raft_consensus.cc:1261] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 33643294a0eb433e8931f67583d665e8 [term 1 FOLLOWER]: Refusing update from remote peer 82c52234a3a94312aae42cfb5ab3ff58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:49:27.870079 24683 raft_consensus.cc:1261] T 7d81d3e2a8fc47c49e7d8cd4378844ff P df6cca547d3847d08cf92cec310077c8 [term 1 FOLLOWER]: Refusing update from remote peer 82c52234a3a94312aae42cfb5ab3ff58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:49:27.870329 24929 consensus_queue.cc:1035] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 82c52234a3a94312aae42cfb5ab3ff58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:49:27.870631 24901 consensus_queue.cc:1035] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 82c52234a3a94312aae42cfb5ab3ff58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:49:27.872464 24947 mvcc.cc:205] Tried to move back new op lower bound from 7334444105194033152 to 7334444103063322624. Current Snapshot: MvccSnapshot[applied={T|T < 7334444105194033152}]
I20260928 22:49:27.885340 24385 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:49:27.885921 24385 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:49:27.887348 24385 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 20a5b16187d042b5ac6e0609f080bced: Sending DeleteTablet for 3 replicas of tablet 7d81d3e2a8fc47c49e7d8cd4378844ff
I20260928 22:49:27.887796 24663 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d81d3e2a8fc47c49e7d8cd4378844ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:49:27 UTC) from {username='oryx', principal='oryx/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:53028
I20260928 22:49:27.887879 24523 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d81d3e2a8fc47c49e7d8cd4378844ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:49:27 UTC) from {username='oryx', principal='oryx/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:52210
I20260928 22:49:27.887998 24802 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d81d3e2a8fc47c49e7d8cd4378844ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:49:27 UTC) from {username='oryx', principal='oryx/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:40098
I20260928 22:49:27.888075 24962 tablet_replica.cc:330] T 7d81d3e2a8fc47c49e7d8cd4378844ff P df6cca547d3847d08cf92cec310077c8: stopping tablet replica
I20260928 22:49:27.888126 24962 raft_consensus.cc:2229] T 7d81d3e2a8fc47c49e7d8cd4378844ff P df6cca547d3847d08cf92cec310077c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:49:27.888211 24962 raft_consensus.cc:2258] T 7d81d3e2a8fc47c49e7d8cd4378844ff P df6cca547d3847d08cf92cec310077c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:49:27.888530 24962 ts_tablet_manager.cc:1905] T 7d81d3e2a8fc47c49e7d8cd4378844ff P df6cca547d3847d08cf92cec310077c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:49:27.888531 24963 tablet_replica.cc:330] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 33643294a0eb433e8931f67583d665e8: stopping tablet replica
I20260928 22:49:27.888600 24963 raft_consensus.cc:2229] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 33643294a0eb433e8931f67583d665e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:49:27.888643 24963 raft_consensus.cc:2258] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 33643294a0eb433e8931f67583d665e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:49:27.888955 24963 ts_tablet_manager.cc:1905] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 33643294a0eb433e8931f67583d665e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:49:27.890743 24964 tablet_replica.cc:330] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 82c52234a3a94312aae42cfb5ab3ff58: stopping tablet replica
I20260928 22:49:27.891008 24964 raft_consensus.cc:2229] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 82c52234a3a94312aae42cfb5ab3ff58 [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:49:27.891281 24964 raft_consensus.cc:2258] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 82c52234a3a94312aae42cfb5ab3ff58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:49:27.891705 24964 ts_tablet_manager.cc:1905] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 82c52234a3a94312aae42cfb5ab3ff58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:49:27.891933 24962 ts_tablet_manager.cc:1918] T 7d81d3e2a8fc47c49e7d8cd4378844ff P df6cca547d3847d08cf92cec310077c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 22:49:27.891991 24962 log.cc:1196] T 7d81d3e2a8fc47c49e7d8cd4378844ff P df6cca547d3847d08cf92cec310077c8: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-1/wal/wals/7d81d3e2a8fc47c49e7d8cd4378844ff
I20260928 22:49:27.892215 24962 ts_tablet_manager.cc:1939] T 7d81d3e2a8fc47c49e7d8cd4378844ff P df6cca547d3847d08cf92cec310077c8: Deleting consensus metadata
I20260928 22:49:27.892380 24963 ts_tablet_manager.cc:1918] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 33643294a0eb433e8931f67583d665e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 22:49:27.892536 24963 log.cc:1196] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 33643294a0eb433e8931f67583d665e8: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-0/wal/wals/7d81d3e2a8fc47c49e7d8cd4378844ff
I20260928 22:49:27.892616 24370 catalog_manager.cc:4841] TS df6cca547d3847d08cf92cec310077c8 (127.16.121.130:38577): tablet 7d81d3e2a8fc47c49e7d8cd4378844ff (table test-table [id=81eb4c9fa9e843448691bb0add1f5539]) successfully deleted
I20260928 22:49:27.893010 24963 ts_tablet_manager.cc:1939] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 33643294a0eb433e8931f67583d665e8: Deleting consensus metadata
I20260928 22:49:27.893646 24370 catalog_manager.cc:4841] TS 33643294a0eb433e8931f67583d665e8 (127.16.121.129:37109): tablet 7d81d3e2a8fc47c49e7d8cd4378844ff (table test-table [id=81eb4c9fa9e843448691bb0add1f5539]) successfully deleted
I20260928 22:49:27.894516 24964 ts_tablet_manager.cc:1918] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 82c52234a3a94312aae42cfb5ab3ff58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 22:49:27.894644 24964 log.cc:1196] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 82c52234a3a94312aae42cfb5ab3ff58: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-2/wal/wals/7d81d3e2a8fc47c49e7d8cd4378844ff
I20260928 22:49:27.894936 24964 ts_tablet_manager.cc:1939] T 7d81d3e2a8fc47c49e7d8cd4378844ff P 82c52234a3a94312aae42cfb5ab3ff58: Deleting consensus metadata
I20260928 22:49:27.895397 24372 catalog_manager.cc:4841] TS 82c52234a3a94312aae42cfb5ab3ff58 (127.16.121.131:36097): tablet 7d81d3e2a8fc47c49e7d8cd4378844ff (table test-table [id=81eb4c9fa9e843448691bb0add1f5539]) successfully deleted
Sep 28 22:49:27 dist-test-slave-56vw krb5kdc[24341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 22:49:27 dist-test-slave-56vw krb5kdc[24341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:27 dist-test-slave-56vw krb5kdc[24341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
W20260928 22:49:27.920169 24972 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:27.912783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44142 (local address 127.16.121.190:40321)
0928 22:49:27.912922 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:27.912924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:27.913029 (+ 105us) server_negotiation.cc:407] Connection header received
0928 22:49:27.913123 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:27.913124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:27.913135 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:27.913198 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:27.914063 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.914727 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.916034 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.916240 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.916419 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:27.920073 (+ 3654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44142: BlockingRecv error: recv got EOF from 127.0.0.1:44142 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":69,"threads_started":1}
Sep 28 22:49:27 dist-test-slave-56vw krb5kdc[24341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM
I20260928 22:49:27.931535 24972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:27.921328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44158 (local address 127.16.121.190:40321)
0928 22:49:27.921369 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:27.921371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:27.921565 (+ 194us) server_negotiation.cc:407] Connection header received
0928 22:49:27.921600 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:27.921601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:27.921613 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:27.921677 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:27.922379 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.922936 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.924175 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:27.924412 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:27.924477 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:27.927885 (+ 3408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:27.927887 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:27.927887 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:27.927898 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:27.930127 (+ 2229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:27.930572 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:27.930574 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:27.930574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:27.930644 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:27.930887 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:27.930888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:27.930888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:27.931030 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:27.931184 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:27.931311 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 22:49:27.931416 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 22:49:27.933815 24385 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44158:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:49:27.934039 24385 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:49:27.936097 24523 tablet_service.cc:1467] Processing CreateTablet for tablet 62c991dc39274730ac4c868a2293ce2a (DEFAULT_TABLE table=test-table [id=1eb29d912c87450e8cdfebee8a07c659]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:49:27.936149 24802 tablet_service.cc:1467] Processing CreateTablet for tablet 62c991dc39274730ac4c868a2293ce2a (DEFAULT_TABLE table=test-table [id=1eb29d912c87450e8cdfebee8a07c659]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:49:27.936177 24663 tablet_service.cc:1467] Processing CreateTablet for tablet 62c991dc39274730ac4c868a2293ce2a (DEFAULT_TABLE table=test-table [id=1eb29d912c87450e8cdfebee8a07c659]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:49:27.936270 24523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62c991dc39274730ac4c868a2293ce2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:27.936283 24663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62c991dc39274730ac4c868a2293ce2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:27.936534 24802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62c991dc39274730ac4c868a2293ce2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:27.937718 24978 tablet_bootstrap.cc:492] T 62c991dc39274730ac4c868a2293ce2a P 33643294a0eb433e8931f67583d665e8: Bootstrap starting.
I20260928 22:49:27.937968 24979 tablet_bootstrap.cc:492] T 62c991dc39274730ac4c868a2293ce2a P df6cca547d3847d08cf92cec310077c8: Bootstrap starting.
I20260928 22:49:27.938122 24978 tablet_bootstrap.cc:654] T 62c991dc39274730ac4c868a2293ce2a P 33643294a0eb433e8931f67583d665e8: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:27.938498 24979 tablet_bootstrap.cc:654] T 62c991dc39274730ac4c868a2293ce2a P df6cca547d3847d08cf92cec310077c8: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:27.938627 24981 tablet_bootstrap.cc:492] T 62c991dc39274730ac4c868a2293ce2a P 82c52234a3a94312aae42cfb5ab3ff58: Bootstrap starting.
I20260928 22:49:27.939105 24981 tablet_bootstrap.cc:654] T 62c991dc39274730ac4c868a2293ce2a P 82c52234a3a94312aae42cfb5ab3ff58: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:27.939308 24979 tablet_bootstrap.cc:492] T 62c991dc39274730ac4c868a2293ce2a P df6cca547d3847d08cf92cec310077c8: No bootstrap required, opened a new log
I20260928 22:49:27.939355 24979 ts_tablet_manager.cc:1397] T 62c991dc39274730ac4c868a2293ce2a P df6cca547d3847d08cf92cec310077c8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:27.939532 24979 raft_consensus.cc:348] T 62c991dc39274730ac4c868a2293ce2a P df6cca547d3847d08cf92cec310077c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } }
I20260928 22:49:27.939610 24979 raft_consensus.cc:374] T 62c991dc39274730ac4c868a2293ce2a P df6cca547d3847d08cf92cec310077c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:27.939608 24981 tablet_bootstrap.cc:492] T 62c991dc39274730ac4c868a2293ce2a P 82c52234a3a94312aae42cfb5ab3ff58: No bootstrap required, opened a new log
I20260928 22:49:27.939634 24979 raft_consensus.cc:729] T 62c991dc39274730ac4c868a2293ce2a P df6cca547d3847d08cf92cec310077c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6cca547d3847d08cf92cec310077c8, State: Initialized, Role: FOLLOWER
I20260928 22:49:27.939651 24981 ts_tablet_manager.cc:1397] T 62c991dc39274730ac4c868a2293ce2a P 82c52234a3a94312aae42cfb5ab3ff58: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 22:49:27.939707 24979 consensus_queue.cc:260] T 62c991dc39274730ac4c868a2293ce2a P df6cca547d3847d08cf92cec310077c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } }
I20260928 22:49:27.939805 24979 ts_tablet_manager.cc:1428] T 62c991dc39274730ac4c868a2293ce2a P df6cca547d3847d08cf92cec310077c8: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 22:49:27.939777 24981 raft_consensus.cc:348] T 62c991dc39274730ac4c868a2293ce2a P 82c52234a3a94312aae42cfb5ab3ff58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } }
I20260928 22:49:27.939842 24981 raft_consensus.cc:374] T 62c991dc39274730ac4c868a2293ce2a P 82c52234a3a94312aae42cfb5ab3ff58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:27.939867 24981 raft_consensus.cc:729] T 62c991dc39274730ac4c868a2293ce2a P 82c52234a3a94312aae42cfb5ab3ff58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82c52234a3a94312aae42cfb5ab3ff58, State: Initialized, Role: FOLLOWER
I20260928 22:49:27.939925 24981 consensus_queue.cc:260] T 62c991dc39274730ac4c868a2293ce2a P 82c52234a3a94312aae42cfb5ab3ff58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } }
I20260928 22:49:27.940030 24981 ts_tablet_manager.cc:1428] T 62c991dc39274730ac4c868a2293ce2a P 82c52234a3a94312aae42cfb5ab3ff58: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:27.940263 24978 tablet_bootstrap.cc:492] T 62c991dc39274730ac4c868a2293ce2a P 33643294a0eb433e8931f67583d665e8: No bootstrap required, opened a new log
I20260928 22:49:27.940316 24978 ts_tablet_manager.cc:1397] T 62c991dc39274730ac4c868a2293ce2a P 33643294a0eb433e8931f67583d665e8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 22:49:27.940451 24978 raft_consensus.cc:348] T 62c991dc39274730ac4c868a2293ce2a P 33643294a0eb433e8931f67583d665e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } }
I20260928 22:49:27.940536 24978 raft_consensus.cc:374] T 62c991dc39274730ac4c868a2293ce2a P 33643294a0eb433e8931f67583d665e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:27.940563 24978 raft_consensus.cc:729] T 62c991dc39274730ac4c868a2293ce2a P 33643294a0eb433e8931f67583d665e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33643294a0eb433e8931f67583d665e8, State: Initialized, Role: FOLLOWER
I20260928 22:49:27.940613 24978 consensus_queue.cc:260] T 62c991dc39274730ac4c868a2293ce2a P 33643294a0eb433e8931f67583d665e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } }
I20260928 22:49:27.940707 24978 ts_tablet_manager.cc:1428] T 62c991dc39274730ac4c868a2293ce2a P 33643294a0eb433e8931f67583d665e8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:28.193231 24915 raft_consensus.cc:482] T 62c991dc39274730ac4c868a2293ce2a P 33643294a0eb433e8931f67583d665e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:49:28.193365 24915 raft_consensus.cc:504] T 62c991dc39274730ac4c868a2293ce2a P 33643294a0eb433e8931f67583d665e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } }
I20260928 22:49:28.193598 24915 leader_election.cc:288] T 62c991dc39274730ac4c868a2293ce2a P 33643294a0eb433e8931f67583d665e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df6cca547d3847d08cf92cec310077c8 (127.16.121.130:38577), 82c52234a3a94312aae42cfb5ab3ff58 (127.16.121.131:36097)
I20260928 22:49:28.193964 24823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62c991dc39274730ac4c868a2293ce2a" candidate_uuid: "33643294a0eb433e8931f67583d665e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82c52234a3a94312aae42cfb5ab3ff58" is_pre_election: true
I20260928 22:49:28.194005 24683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62c991dc39274730ac4c868a2293ce2a" candidate_uuid: "33643294a0eb433e8931f67583d665e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6cca547d3847d08cf92cec310077c8" is_pre_election: true
I20260928 22:49:28.194087 24823 raft_consensus.cc:2454] T 62c991dc39274730ac4c868a2293ce2a P 82c52234a3a94312aae42cfb5ab3ff58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33643294a0eb433e8931f67583d665e8 in term 0.
I20260928 22:49:28.194088 24683 raft_consensus.cc:2454] T 62c991dc39274730ac4c868a2293ce2a P df6cca547d3847d08cf92cec310077c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33643294a0eb433e8931f67583d665e8 in term 0.
I20260928 22:49:28.194280 24474 leader_election.cc:302] T 62c991dc39274730ac4c868a2293ce2a P 33643294a0eb433e8931f67583d665e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33643294a0eb433e8931f67583d665e8, df6cca547d3847d08cf92cec310077c8; no voters:
I20260928 22:49:28.194635 24915 raft_consensus.cc:2781] T 62c991dc39274730ac4c868a2293ce2a P 33643294a0eb433e8931f67583d665e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:49:28.194705 24915 raft_consensus.cc:482] T 62c991dc39274730ac4c868a2293ce2a P 33643294a0eb433e8931f67583d665e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:49:28.194725 24915 raft_consensus.cc:3037] T 62c991dc39274730ac4c868a2293ce2a P 33643294a0eb433e8931f67583d665e8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:28.195691 24915 raft_consensus.cc:504] T 62c991dc39274730ac4c868a2293ce2a P 33643294a0eb433e8931f67583d665e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } }
I20260928 22:49:28.195905 24915 leader_election.cc:288] T 62c991dc39274730ac4c868a2293ce2a P 33643294a0eb433e8931f67583d665e8 [CANDIDATE]: Term 1 election: Requested vote from peers df6cca547d3847d08cf92cec310077c8 (127.16.121.130:38577), 82c52234a3a94312aae42cfb5ab3ff58 (127.16.121.131:36097)
I20260928 22:49:28.196144 24683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62c991dc39274730ac4c868a2293ce2a" candidate_uuid: "33643294a0eb433e8931f67583d665e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6cca547d3847d08cf92cec310077c8"
I20260928 22:49:28.196250 24683 raft_consensus.cc:3037] T 62c991dc39274730ac4c868a2293ce2a P df6cca547d3847d08cf92cec310077c8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:28.196262 24823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62c991dc39274730ac4c868a2293ce2a" candidate_uuid: "33643294a0eb433e8931f67583d665e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82c52234a3a94312aae42cfb5ab3ff58"
I20260928 22:49:28.196391 24823 raft_consensus.cc:3037] T 62c991dc39274730ac4c868a2293ce2a P 82c52234a3a94312aae42cfb5ab3ff58 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:28.196975 24683 raft_consensus.cc:2454] T 62c991dc39274730ac4c868a2293ce2a P df6cca547d3847d08cf92cec310077c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33643294a0eb433e8931f67583d665e8 in term 1.
I20260928 22:49:28.197108 24823 raft_consensus.cc:2454] T 62c991dc39274730ac4c868a2293ce2a P 82c52234a3a94312aae42cfb5ab3ff58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33643294a0eb433e8931f67583d665e8 in term 1.
I20260928 22:49:28.197238 24474 leader_election.cc:302] T 62c991dc39274730ac4c868a2293ce2a P 33643294a0eb433e8931f67583d665e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33643294a0eb433e8931f67583d665e8, df6cca547d3847d08cf92cec310077c8; no voters:
I20260928 22:49:28.197359 24915 raft_consensus.cc:2781] T 62c991dc39274730ac4c868a2293ce2a P 33643294a0eb433e8931f67583d665e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:49:28.197466 24915 raft_consensus.cc:686] T 62c991dc39274730ac4c868a2293ce2a P 33643294a0eb433e8931f67583d665e8 [term 1 LEADER]: Becoming Leader. State: Replica: 33643294a0eb433e8931f67583d665e8, State: Running, Role: LEADER
I20260928 22:49:28.197508 24915 consensus_queue.cc:237] T 62c991dc39274730ac4c868a2293ce2a P 33643294a0eb433e8931f67583d665e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } }
I20260928 22:49:28.198416 24385 catalog_manager.cc:5495] T 62c991dc39274730ac4c868a2293ce2a P 33643294a0eb433e8931f67583d665e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33643294a0eb433e8931f67583d665e8 (127.16.121.129). New cstate: current_term: 1 leader_uuid: "33643294a0eb433e8931f67583d665e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } health_report { overall_health: UNKNOWN } } }
I20260928 22:49:28.232815 24987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:28.228377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55262 (local address 127.16.121.129:37109)
0928 22:49:28.228706 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:28.228709 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:28.228718 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:49:28.228745 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:28.228745 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:28.228756 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:28.228966 (+ 210us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:49:28.229223 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:28.230422 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:28.231550 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:28.231808 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:28.231893 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:28.232223 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:49:28.232260 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:49:28.232319 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:28.232624 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 22:49:28.232654 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":115,"threads_started":1}
I20260928 22:49:28.234215 24823 raft_consensus.cc:1261] T 62c991dc39274730ac4c868a2293ce2a P 82c52234a3a94312aae42cfb5ab3ff58 [term 1 FOLLOWER]: Refusing update from remote peer 33643294a0eb433e8931f67583d665e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:49:28.234496 24683 raft_consensus.cc:1261] T 62c991dc39274730ac4c868a2293ce2a P df6cca547d3847d08cf92cec310077c8 [term 1 FOLLOWER]: Refusing update from remote peer 33643294a0eb433e8931f67583d665e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:49:28.234584 24915 consensus_queue.cc:1035] T 62c991dc39274730ac4c868a2293ce2a P 33643294a0eb433e8931f67583d665e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:49:28.234778 24915 consensus_queue.cc:1035] T 62c991dc39274730ac4c868a2293ce2a P 33643294a0eb433e8931f67583d665e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:49:28.239413 24385 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:49:28.239562 24385 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:49:28.240274 24385 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 20a5b16187d042b5ac6e0609f080bced: Sending DeleteTablet for 3 replicas of tablet 62c991dc39274730ac4c868a2293ce2a
I20260928 22:49:28.240689 24523 tablet_service.cc:1514] Processing DeleteTablet for tablet 62c991dc39274730ac4c868a2293ce2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:49:28 UTC) from {username='oryx', principal='oryx/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:52210
I20260928 22:49:28.240697 24663 tablet_service.cc:1514] Processing DeleteTablet for tablet 62c991dc39274730ac4c868a2293ce2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:49:28 UTC) from {username='oryx', principal='oryx/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:53028
I20260928 22:49:28.240808 24802 tablet_service.cc:1514] Processing DeleteTablet for tablet 62c991dc39274730ac4c868a2293ce2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:49:28 UTC) from {username='oryx', principal='oryx/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:40098
I20260928 22:49:28.240916 24962 tablet_replica.cc:330] T 62c991dc39274730ac4c868a2293ce2a P df6cca547d3847d08cf92cec310077c8: stopping tablet replica
I20260928 22:49:28.240936 24964 tablet_replica.cc:330] T 62c991dc39274730ac4c868a2293ce2a P 82c52234a3a94312aae42cfb5ab3ff58: stopping tablet replica
I20260928 22:49:28.240947 24963 tablet_replica.cc:330] T 62c991dc39274730ac4c868a2293ce2a P 33643294a0eb433e8931f67583d665e8: stopping tablet replica
I20260928 22:49:28.240986 24962 raft_consensus.cc:2229] T 62c991dc39274730ac4c868a2293ce2a P df6cca547d3847d08cf92cec310077c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:49:28.240991 24964 raft_consensus.cc:2229] T 62c991dc39274730ac4c868a2293ce2a P 82c52234a3a94312aae42cfb5ab3ff58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:49:28.241035 24963 raft_consensus.cc:2229] T 62c991dc39274730ac4c868a2293ce2a P 33643294a0eb433e8931f67583d665e8 [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:49:28.241042 24962 raft_consensus.cc:2258] T 62c991dc39274730ac4c868a2293ce2a P df6cca547d3847d08cf92cec310077c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:49:28.241053 24964 raft_consensus.cc:2258] T 62c991dc39274730ac4c868a2293ce2a P 82c52234a3a94312aae42cfb5ab3ff58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:49:28.241115 24963 raft_consensus.cc:2258] T 62c991dc39274730ac4c868a2293ce2a P 33643294a0eb433e8931f67583d665e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:49:28.241509 24964 ts_tablet_manager.cc:1905] T 62c991dc39274730ac4c868a2293ce2a P 82c52234a3a94312aae42cfb5ab3ff58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:49:28.241393 24385 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44158:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 22:49:28.241590 24962 ts_tablet_manager.cc:1905] T 62c991dc39274730ac4c868a2293ce2a P df6cca547d3847d08cf92cec310077c8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 22:49:28.241654 24385 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:49:28.242117 24963 ts_tablet_manager.cc:1905] T 62c991dc39274730ac4c868a2293ce2a P 33643294a0eb433e8931f67583d665e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:49:28.243656 24962 ts_tablet_manager.cc:1918] T 62c991dc39274730ac4c868a2293ce2a P df6cca547d3847d08cf92cec310077c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:49:28.243651 24802 tablet_service.cc:1467] Processing CreateTablet for tablet 87d1cb48e7f84e739b875b928cf52eed (DEFAULT_TABLE table=test-table [id=c811d4383f1d4b108c91a52f3dfff7b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:49:28.243738 24962 log.cc:1196] T 62c991dc39274730ac4c868a2293ce2a P df6cca547d3847d08cf92cec310077c8: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-1/wal/wals/62c991dc39274730ac4c868a2293ce2a
I20260928 22:49:28.243922 24523 tablet_service.cc:1467] Processing CreateTablet for tablet 87d1cb48e7f84e739b875b928cf52eed (DEFAULT_TABLE table=test-table [id=c811d4383f1d4b108c91a52f3dfff7b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:49:28.243976 24962 ts_tablet_manager.cc:1939] T 62c991dc39274730ac4c868a2293ce2a P df6cca547d3847d08cf92cec310077c8: Deleting consensus metadata
I20260928 22:49:28.244069 24964 ts_tablet_manager.cc:1918] T 62c991dc39274730ac4c868a2293ce2a P 82c52234a3a94312aae42cfb5ab3ff58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:49:28.244088 24523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87d1cb48e7f84e739b875b928cf52eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:28.244123 24964 log.cc:1196] T 62c991dc39274730ac4c868a2293ce2a P 82c52234a3a94312aae42cfb5ab3ff58: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-2/wal/wals/62c991dc39274730ac4c868a2293ce2a
I20260928 22:49:28.244326 24964 ts_tablet_manager.cc:1939] T 62c991dc39274730ac4c868a2293ce2a P 82c52234a3a94312aae42cfb5ab3ff58: Deleting consensus metadata
I20260928 22:49:28.244495 24802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87d1cb48e7f84e739b875b928cf52eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:28.244735 24372 catalog_manager.cc:4841] TS 82c52234a3a94312aae42cfb5ab3ff58 (127.16.121.131:36097): tablet 62c991dc39274730ac4c868a2293ce2a (table test-table [id=1eb29d912c87450e8cdfebee8a07c659]) successfully deleted
I20260928 22:49:28.245287 24978 tablet_bootstrap.cc:492] T 87d1cb48e7f84e739b875b928cf52eed P 33643294a0eb433e8931f67583d665e8: Bootstrap starting.
I20260928 22:49:28.245494 24663 tablet_service.cc:1467] Processing CreateTablet for tablet 87d1cb48e7f84e739b875b928cf52eed (DEFAULT_TABLE table=test-table [id=c811d4383f1d4b108c91a52f3dfff7b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:49:28.245512 24370 catalog_manager.cc:4841] TS df6cca547d3847d08cf92cec310077c8 (127.16.121.130:38577): tablet 62c991dc39274730ac4c868a2293ce2a (table test-table [id=1eb29d912c87450e8cdfebee8a07c659]) successfully deleted
I20260928 22:49:28.245596 24663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87d1cb48e7f84e739b875b928cf52eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:28.245608 24981 tablet_bootstrap.cc:492] T 87d1cb48e7f84e739b875b928cf52eed P 82c52234a3a94312aae42cfb5ab3ff58: Bootstrap starting.
I20260928 22:49:28.245694 24978 tablet_bootstrap.cc:654] T 87d1cb48e7f84e739b875b928cf52eed P 33643294a0eb433e8931f67583d665e8: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:28.243953 24963 ts_tablet_manager.cc:1918] T 62c991dc39274730ac4c868a2293ce2a P 33643294a0eb433e8931f67583d665e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:49:28.245893 24963 log.cc:1196] T 62c991dc39274730ac4c868a2293ce2a P 33643294a0eb433e8931f67583d665e8: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-0/wal/wals/62c991dc39274730ac4c868a2293ce2a
I20260928 22:49:28.245899 24981 tablet_bootstrap.cc:654] T 87d1cb48e7f84e739b875b928cf52eed P 82c52234a3a94312aae42cfb5ab3ff58: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:28.246140 24963 ts_tablet_manager.cc:1939] T 62c991dc39274730ac4c868a2293ce2a P 33643294a0eb433e8931f67583d665e8: Deleting consensus metadata
I20260928 22:49:28.246518 24370 catalog_manager.cc:4841] TS 33643294a0eb433e8931f67583d665e8 (127.16.121.129:37109): tablet 62c991dc39274730ac4c868a2293ce2a (table test-table [id=1eb29d912c87450e8cdfebee8a07c659]) successfully deleted
I20260928 22:49:28.246555 24981 tablet_bootstrap.cc:492] T 87d1cb48e7f84e739b875b928cf52eed P 82c52234a3a94312aae42cfb5ab3ff58: No bootstrap required, opened a new log
I20260928 22:49:28.246598 24981 ts_tablet_manager.cc:1397] T 87d1cb48e7f84e739b875b928cf52eed P 82c52234a3a94312aae42cfb5ab3ff58: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 22:49:28.246744 24978 tablet_bootstrap.cc:492] T 87d1cb48e7f84e739b875b928cf52eed P 33643294a0eb433e8931f67583d665e8: No bootstrap required, opened a new log
I20260928 22:49:28.246719 24981 raft_consensus.cc:348] T 87d1cb48e7f84e739b875b928cf52eed P 82c52234a3a94312aae42cfb5ab3ff58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } }
I20260928 22:49:28.246780 24978 ts_tablet_manager.cc:1397] T 87d1cb48e7f84e739b875b928cf52eed P 33643294a0eb433e8931f67583d665e8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:49:28.246793 24981 raft_consensus.cc:374] T 87d1cb48e7f84e739b875b928cf52eed P 82c52234a3a94312aae42cfb5ab3ff58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:28.246815 24981 raft_consensus.cc:729] T 87d1cb48e7f84e739b875b928cf52eed P 82c52234a3a94312aae42cfb5ab3ff58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82c52234a3a94312aae42cfb5ab3ff58, State: Initialized, Role: FOLLOWER
I20260928 22:49:28.246873 24981 consensus_queue.cc:260] T 87d1cb48e7f84e739b875b928cf52eed P 82c52234a3a94312aae42cfb5ab3ff58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } }
I20260928 22:49:28.246912 24978 raft_consensus.cc:348] T 87d1cb48e7f84e739b875b928cf52eed P 33643294a0eb433e8931f67583d665e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } }
I20260928 22:49:28.246979 24981 ts_tablet_manager.cc:1428] T 87d1cb48e7f84e739b875b928cf52eed P 82c52234a3a94312aae42cfb5ab3ff58: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 22:49:28.246968 24979 tablet_bootstrap.cc:492] T 87d1cb48e7f84e739b875b928cf52eed P df6cca547d3847d08cf92cec310077c8: Bootstrap starting.
I20260928 22:49:28.246989 24978 raft_consensus.cc:374] T 87d1cb48e7f84e739b875b928cf52eed P 33643294a0eb433e8931f67583d665e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:28.247016 24978 raft_consensus.cc:729] T 87d1cb48e7f84e739b875b928cf52eed P 33643294a0eb433e8931f67583d665e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33643294a0eb433e8931f67583d665e8, State: Initialized, Role: FOLLOWER
I20260928 22:49:28.247061 24978 consensus_queue.cc:260] T 87d1cb48e7f84e739b875b928cf52eed P 33643294a0eb433e8931f67583d665e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } }
I20260928 22:49:28.247203 24978 ts_tablet_manager.cc:1428] T 87d1cb48e7f84e739b875b928cf52eed P 33643294a0eb433e8931f67583d665e8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:28.247462 24979 tablet_bootstrap.cc:654] T 87d1cb48e7f84e739b875b928cf52eed P df6cca547d3847d08cf92cec310077c8: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:28.248265 24979 tablet_bootstrap.cc:492] T 87d1cb48e7f84e739b875b928cf52eed P df6cca547d3847d08cf92cec310077c8: No bootstrap required, opened a new log
I20260928 22:49:28.248335 24979 ts_tablet_manager.cc:1397] T 87d1cb48e7f84e739b875b928cf52eed P df6cca547d3847d08cf92cec310077c8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:28.248490 24979 raft_consensus.cc:348] T 87d1cb48e7f84e739b875b928cf52eed P df6cca547d3847d08cf92cec310077c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } }
I20260928 22:49:28.248567 24979 raft_consensus.cc:374] T 87d1cb48e7f84e739b875b928cf52eed P df6cca547d3847d08cf92cec310077c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:28.248584 24979 raft_consensus.cc:729] T 87d1cb48e7f84e739b875b928cf52eed P df6cca547d3847d08cf92cec310077c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6cca547d3847d08cf92cec310077c8, State: Initialized, Role: FOLLOWER
I20260928 22:49:28.248644 24979 consensus_queue.cc:260] T 87d1cb48e7f84e739b875b928cf52eed P df6cca547d3847d08cf92cec310077c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } }
I20260928 22:49:28.248744 24979 ts_tablet_manager.cc:1428] T 87d1cb48e7f84e739b875b928cf52eed P df6cca547d3847d08cf92cec310077c8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:28.275202 24990 raft_consensus.cc:482] T 87d1cb48e7f84e739b875b928cf52eed P 33643294a0eb433e8931f67583d665e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:49:28.275358 24990 raft_consensus.cc:504] T 87d1cb48e7f84e739b875b928cf52eed P 33643294a0eb433e8931f67583d665e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } }
I20260928 22:49:28.275545 24990 leader_election.cc:288] T 87d1cb48e7f84e739b875b928cf52eed P 33643294a0eb433e8931f67583d665e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df6cca547d3847d08cf92cec310077c8 (127.16.121.130:38577), 82c52234a3a94312aae42cfb5ab3ff58 (127.16.121.131:36097)
I20260928 22:49:28.275794 24683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87d1cb48e7f84e739b875b928cf52eed" candidate_uuid: "33643294a0eb433e8931f67583d665e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6cca547d3847d08cf92cec310077c8" is_pre_election: true
I20260928 22:49:28.275795 24823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87d1cb48e7f84e739b875b928cf52eed" candidate_uuid: "33643294a0eb433e8931f67583d665e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82c52234a3a94312aae42cfb5ab3ff58" is_pre_election: true
I20260928 22:49:28.275941 24683 raft_consensus.cc:2454] T 87d1cb48e7f84e739b875b928cf52eed P df6cca547d3847d08cf92cec310077c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33643294a0eb433e8931f67583d665e8 in term 0.
I20260928 22:49:28.275959 24823 raft_consensus.cc:2454] T 87d1cb48e7f84e739b875b928cf52eed P 82c52234a3a94312aae42cfb5ab3ff58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33643294a0eb433e8931f67583d665e8 in term 0.
I20260928 22:49:28.276155 24476 leader_election.cc:302] T 87d1cb48e7f84e739b875b928cf52eed P 33643294a0eb433e8931f67583d665e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33643294a0eb433e8931f67583d665e8, 82c52234a3a94312aae42cfb5ab3ff58; no voters:
I20260928 22:49:28.276317 24990 raft_consensus.cc:2781] T 87d1cb48e7f84e739b875b928cf52eed P 33643294a0eb433e8931f67583d665e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:49:28.276378 24990 raft_consensus.cc:482] T 87d1cb48e7f84e739b875b928cf52eed P 33643294a0eb433e8931f67583d665e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:49:28.276396 24990 raft_consensus.cc:3037] T 87d1cb48e7f84e739b875b928cf52eed P 33643294a0eb433e8931f67583d665e8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:28.277177 24990 raft_consensus.cc:504] T 87d1cb48e7f84e739b875b928cf52eed P 33643294a0eb433e8931f67583d665e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } }
I20260928 22:49:28.277352 24990 leader_election.cc:288] T 87d1cb48e7f84e739b875b928cf52eed P 33643294a0eb433e8931f67583d665e8 [CANDIDATE]: Term 1 election: Requested vote from peers df6cca547d3847d08cf92cec310077c8 (127.16.121.130:38577), 82c52234a3a94312aae42cfb5ab3ff58 (127.16.121.131:36097)
I20260928 22:49:28.277529 24683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87d1cb48e7f84e739b875b928cf52eed" candidate_uuid: "33643294a0eb433e8931f67583d665e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6cca547d3847d08cf92cec310077c8"
I20260928 22:49:28.277531 24823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87d1cb48e7f84e739b875b928cf52eed" candidate_uuid: "33643294a0eb433e8931f67583d665e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82c52234a3a94312aae42cfb5ab3ff58"
I20260928 22:49:28.277618 24683 raft_consensus.cc:3037] T 87d1cb48e7f84e739b875b928cf52eed P df6cca547d3847d08cf92cec310077c8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:28.277623 24823 raft_consensus.cc:3037] T 87d1cb48e7f84e739b875b928cf52eed P 82c52234a3a94312aae42cfb5ab3ff58 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:28.278496 24683 raft_consensus.cc:2454] T 87d1cb48e7f84e739b875b928cf52eed P df6cca547d3847d08cf92cec310077c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33643294a0eb433e8931f67583d665e8 in term 1.
I20260928 22:49:28.278491 24823 raft_consensus.cc:2454] T 87d1cb48e7f84e739b875b928cf52eed P 82c52234a3a94312aae42cfb5ab3ff58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33643294a0eb433e8931f67583d665e8 in term 1.
I20260928 22:49:28.278748 24476 leader_election.cc:302] T 87d1cb48e7f84e739b875b928cf52eed P 33643294a0eb433e8931f67583d665e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33643294a0eb433e8931f67583d665e8, 82c52234a3a94312aae42cfb5ab3ff58; no voters:
I20260928 22:49:28.278851 24990 raft_consensus.cc:2781] T 87d1cb48e7f84e739b875b928cf52eed P 33643294a0eb433e8931f67583d665e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:49:28.278929 24990 raft_consensus.cc:686] T 87d1cb48e7f84e739b875b928cf52eed P 33643294a0eb433e8931f67583d665e8 [term 1 LEADER]: Becoming Leader. State: Replica: 33643294a0eb433e8931f67583d665e8, State: Running, Role: LEADER
I20260928 22:49:28.279007 24990 consensus_queue.cc:237] T 87d1cb48e7f84e739b875b928cf52eed P 33643294a0eb433e8931f67583d665e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } }
I20260928 22:49:28.279738 24386 catalog_manager.cc:5495] T 87d1cb48e7f84e739b875b928cf52eed P 33643294a0eb433e8931f67583d665e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33643294a0eb433e8931f67583d665e8 (127.16.121.129). New cstate: current_term: 1 leader_uuid: "33643294a0eb433e8931f67583d665e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33643294a0eb433e8931f67583d665e8" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 } health_report { overall_health: UNKNOWN } } }
I20260928 22:49:28.298185 24894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:28.294755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:50167 (local address 127.16.121.131:36097)
0928 22:49:28.294861 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:28.294864 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:28.294968 (+ 104us) server_negotiation.cc:407] Connection header received
0928 22:49:28.295086 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:28.295088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:28.295101 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:28.295184 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:49:28.295470 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:28.296116 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:28.297046 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:28.297346 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:28.297441 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:28.297667 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:49:28.297710 (+ 43us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:49:28.297882 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:28.298067 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 22:49:28.298092 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 22:49:28.298221 24997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:28.294660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.131:36097 (local address 127.16.121.129:50167)
0928 22:49:28.294880 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:28.294893 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:28.294988 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:28.295190 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:28.295191 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:28.295193 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 22:49:28.295277 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:28.295278 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:28.296221 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:28.296222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:28.296913 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:28.296917 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:28.297464 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:28.297465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:28.297549 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:28.297569 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 22:49:28.297885 (+ 316us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 22:49:28.297889 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:49:28.298057 (+ 168us) client_negotiation.cc:240] Negotiation successful
0928 22:49:28.298129 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":97,"threads_started":1}
I20260928 22:49:28.301122 24823 raft_consensus.cc:1261] T 87d1cb48e7f84e739b875b928cf52eed P 82c52234a3a94312aae42cfb5ab3ff58 [term 1 FOLLOWER]: Refusing update from remote peer 33643294a0eb433e8931f67583d665e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:49:28.301157 24683 raft_consensus.cc:1261] T 87d1cb48e7f84e739b875b928cf52eed P df6cca547d3847d08cf92cec310077c8 [term 1 FOLLOWER]: Refusing update from remote peer 33643294a0eb433e8931f67583d665e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:49:28.301429 24984 consensus_queue.cc:1035] T 87d1cb48e7f84e739b875b928cf52eed P 33643294a0eb433e8931f67583d665e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6cca547d3847d08cf92cec310077c8" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:49:28.301595 24990 consensus_queue.cc:1035] T 87d1cb48e7f84e739b875b928cf52eed P 33643294a0eb433e8931f67583d665e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82c52234a3a94312aae42cfb5ab3ff58" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 36097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:49:28.310886 24987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:28.307401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:38575 (local address 127.16.121.129:37109)
0928 22:49:28.307470 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:28.307472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:28.307482 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:49:28.307590 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:28.307591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:28.307602 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:28.307657 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:49:28.308019 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:28.308936 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:28.310064 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:28.310319 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:28.310391 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:28.310610 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:49:28.310649 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:49:28.310711 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:28.310796 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 22:49:28.310819 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 22:49:28.310909 24956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:28.307338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:37109 (local address 127.16.121.131:38575)
0928 22:49:28.307420 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:28.307435 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:28.307519 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:28.307684 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:28.307685 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:28.307687 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 22:49:28.307757 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:28.307759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:28.309144 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:28.309145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:28.309975 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:28.309978 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:28.310401 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:28.310401 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:28.310475 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:28.310484 (+ 9us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 22:49:28.310735 (+ 251us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 22:49:28.310738 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:49:28.310792 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 22:49:28.310848 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 22:49:28.321550 24386 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:49:28.321676 24386 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:49:28.322809 24386 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 20a5b16187d042b5ac6e0609f080bced: Sending DeleteTablet for 3 replicas of tablet 87d1cb48e7f84e739b875b928cf52eed
I20260928 22:49:28.323247 24523 tablet_service.cc:1514] Processing DeleteTablet for tablet 87d1cb48e7f84e739b875b928cf52eed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:49:28 UTC) from {username='oryx', principal='oryx/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:52210
I20260928 22:49:28.323274 24802 tablet_service.cc:1514] Processing DeleteTablet for tablet 87d1cb48e7f84e739b875b928cf52eed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:49:28 UTC) from {username='oryx', principal='oryx/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:40098
I20260928 22:49:28.323275 24663 tablet_service.cc:1514] Processing DeleteTablet for tablet 87d1cb48e7f84e739b875b928cf52eed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:49:28 UTC) from {username='oryx', principal='oryx/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:53028
I20260928 22:49:28.323395 24963 tablet_replica.cc:330] T 87d1cb48e7f84e739b875b928cf52eed P 33643294a0eb433e8931f67583d665e8: stopping tablet replica
I20260928 22:49:28.323455 24964 tablet_replica.cc:330] T 87d1cb48e7f84e739b875b928cf52eed P 82c52234a3a94312aae42cfb5ab3ff58: stopping tablet replica
I20260928 22:49:28.323513 24964 raft_consensus.cc:2229] T 87d1cb48e7f84e739b875b928cf52eed P 82c52234a3a94312aae42cfb5ab3ff58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:49:28.323453 24963 raft_consensus.cc:2229] T 87d1cb48e7f84e739b875b928cf52eed P 33643294a0eb433e8931f67583d665e8 [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:49:28.323655 24964 raft_consensus.cc:2258] T 87d1cb48e7f84e739b875b928cf52eed P 82c52234a3a94312aae42cfb5ab3ff58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:49:28.323710 24963 raft_consensus.cc:2258] T 87d1cb48e7f84e739b875b928cf52eed P 33643294a0eb433e8931f67583d665e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:49:28.323758 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 24458
I20260928 22:49:28.323777 24962 tablet_replica.cc:330] T 87d1cb48e7f84e739b875b928cf52eed P df6cca547d3847d08cf92cec310077c8: stopping tablet replica
I20260928 22:49:28.323850 24962 raft_consensus.cc:2229] T 87d1cb48e7f84e739b875b928cf52eed P df6cca547d3847d08cf92cec310077c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:49:28.323891 24962 raft_consensus.cc:2258] T 87d1cb48e7f84e739b875b928cf52eed P df6cca547d3847d08cf92cec310077c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:49:28.324698 24964 ts_tablet_manager.cc:1905] T 87d1cb48e7f84e739b875b928cf52eed P 82c52234a3a94312aae42cfb5ab3ff58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:49:28.327157 24964 ts_tablet_manager.cc:1918] T 87d1cb48e7f84e739b875b928cf52eed P 82c52234a3a94312aae42cfb5ab3ff58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 22:49:28.327247 24964 log.cc:1196] T 87d1cb48e7f84e739b875b928cf52eed P 82c52234a3a94312aae42cfb5ab3ff58: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790635733329745-16870-0/minicluster-data/ts-2/wal/wals/87d1cb48e7f84e739b875b928cf52eed
I20260928 22:49:28.327607 24964 ts_tablet_manager.cc:1939] T 87d1cb48e7f84e739b875b928cf52eed P 82c52234a3a94312aae42cfb5ab3ff58: Deleting consensus metadata
I20260928 22:49:28.328130 24372 catalog_manager.cc:4841] TS 82c52234a3a94312aae42cfb5ab3ff58 (127.16.121.131:36097): tablet 87d1cb48e7f84e739b875b928cf52eed (table test-table [id=c811d4383f1d4b108c91a52f3dfff7b9]) successfully deleted
W20260928 22:49:28.333310 24370 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.121.129:37109 (error 108)
I20260928 22:49:28.333603 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 24598
I20260928 22:49:28.333683 24962 ts_tablet_manager.cc:1905] T 87d1cb48e7f84e739b875b928cf52eed P df6cca547d3847d08cf92cec310077c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:49:28.335330 25001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:28.334939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:37109 (local address 127.0.0.1:55264)
0928 22:49:28.335235 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:28.335286 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.129:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":55,"threads_started":1}
W20260928 22:49:28.335557 24370 catalog_manager.cc:4584] TS 33643294a0eb433e8931f67583d665e8 (127.16.121.129:37109): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 87d1cb48e7f84e739b875b928cf52eed: Network error: Client connection negotiation failed: client connection to 127.16.121.129:37109: connect: Connection refused (error 111)
I20260928 22:49:28.339939 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 24738
I20260928 22:49:28.339988 25001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:28.339651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.130:38577 (local address 127.0.0.1:54404)
0928 22:49:28.339899 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:28.339938 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.130:38577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217}
I20260928 22:49:28.347443 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 24354
2026-09-28T22:49:28Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3267 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790635733329745-16870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:49:28 dist-test-slave-56vw krb5kdc[25004](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:49:28 dist-test-slave-56vw krb5kdc[25004](info): set up 2 sockets
Sep 28 22:49:28 dist-test-slave-56vw krb5kdc[25004](info): commencing operation
krb5kdc: starting...
W20260928 22:49:29.479784 16870 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Sep 28 22:49:29 dist-test-slave-56vw krb5kdc[25004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:49:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:49:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.121.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
I20260928 22:49:29.615295 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:32913
--webserver_interface=127.16.121.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:41081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:32913
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:49:29.701422 25017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:29.701665 25017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:29.701690 25017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:29.702915 25017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:49:29.702958 25017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:29.702973 25017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:29.702986 25017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:49:29.702998 25017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:49:29.704650 25017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:41081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:32913
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:32913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.121.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:29.704873 25017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:29.705061 25017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:29.711028 25023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:29.711084 25025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:29.711128 25017 server_base.cc:1034] running on GCE node
W20260928 22:49:29.710995 25022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:29.711525 25017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:29.711822 25017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:29.713003 25017 hybrid_clock.cc:648] HybridClock initialized: now 1790635769712974 us; error 50 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T22:49:29Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1373 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:49:29 dist-test-slave-56vw krb5kdc[25030](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:49:29 dist-test-slave-56vw krb5kdc[25030](info): set up 2 sockets
Sep 28 22:49:29 dist-test-slave-56vw krb5kdc[25030](info): commencing operation
krb5kdc: starting...
W20260928 22:49:30.863932 16870 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Sep 28 22:49:30 dist-test-slave-56vw krb5kdc[25030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:49:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:49:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.121.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
I20260928 22:49:30.998633 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:44341
--webserver_interface=127.16.121.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:41749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:44341
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:49:31.085081 25043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:31.085320 25043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:31.085345 25043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:31.086612 25043 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:49:31.086663 25043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:31.086678 25043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:31.086690 25043 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:49:31.086702 25043 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:49:31.088336 25043 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:41749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:44341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:44341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.121.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:31.088608 25043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:31.088864 25043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:31.094264 25048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:31.094278 25051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:31.094254 25049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:31.094607 25043 server_base.cc:1034] running on GCE node
I20260928 22:49:31.095017 25043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:31.095938 25043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:31.097239 25043 hybrid_clock.cc:648] HybridClock initialized: now 1790635771097218 us; error 49 us; skew 500 ppm
Sep 28 22:49:31 dist-test-slave-56vw krb5kdc[25030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635771, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:31.108314 25043 init.cc:377] Logged in from keytab as kudu/127.16.121.190@KRBTEST.COM (short username kudu)
I20260928 22:49:31.109170 25043 webserver.cc:460] Webserver started at http://127.16.121.190:43461/ using document root <none> and password file <none>
I20260928 22:49:31.109467 25043 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:31.109551 25043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:31.109738 25043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:31.110898 25043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "08191518230b4f81a094e80efd1424a8"
format_stamp: "Formatted at 2026-09-28 22:49:31 on dist-test-slave-56vw"
server_key: "c9908cae4141884e824ce07982602d26"
server_key_iv: "c6b228d217abdb3b7bf9fcb40d40dcc4"
server_key_version: "encryptionkey@0"
I20260928 22:49:31.111356 25043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "08191518230b4f81a094e80efd1424a8"
format_stamp: "Formatted at 2026-09-28 22:49:31 on dist-test-slave-56vw"
server_key: "c9908cae4141884e824ce07982602d26"
server_key_iv: "c6b228d217abdb3b7bf9fcb40d40dcc4"
server_key_version: "encryptionkey@0"
I20260928 22:49:31.112936 25043 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:31.114006 25058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:31.114303 25043 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:31.114436 25043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "08191518230b4f81a094e80efd1424a8"
format_stamp: "Formatted at 2026-09-28 22:49:31 on dist-test-slave-56vw"
server_key: "c9908cae4141884e824ce07982602d26"
server_key_iv: "c6b228d217abdb3b7bf9fcb40d40dcc4"
server_key_version: "encryptionkey@0"
I20260928 22:49:31.114537 25043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:31.144146 25043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:31.147034 25043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:31.147255 25043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:31.151839 25043 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.190:44341
I20260928 22:49:31.151872 25110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.190:44341 every 8 connection(s)
I20260928 22:49:31.152266 25043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:49:31.153160 25111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:31.154520 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 25043
I20260928 22:49:31.154647 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:49:31.154848 16870 external_mini_cluster.cc:1428] Setting key e3baa6846b6ba264a866ca53a84a070c
I20260928 22:49:31.155937 25111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08191518230b4f81a094e80efd1424a8: Bootstrap starting.
I20260928 22:49:31.156682 25111 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08191518230b4f81a094e80efd1424a8: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:31.157042 25111 log.cc:824] T 00000000000000000000000000000000 P 08191518230b4f81a094e80efd1424a8: Log is configured to *not* fsync() on all Append() calls
I20260928 22:49:31.157785 25111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08191518230b4f81a094e80efd1424a8: No bootstrap required, opened a new log
I20260928 22:49:31.160174 25111 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08191518230b4f81a094e80efd1424a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08191518230b4f81a094e80efd1424a8" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 44341 } }
I20260928 22:49:31.160389 25111 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08191518230b4f81a094e80efd1424a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:31.160419 25111 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08191518230b4f81a094e80efd1424a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08191518230b4f81a094e80efd1424a8, State: Initialized, Role: FOLLOWER
I20260928 22:49:31.160598 25111 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08191518230b4f81a094e80efd1424a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08191518230b4f81a094e80efd1424a8" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 44341 } }
I20260928 22:49:31.160681 25111 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08191518230b4f81a094e80efd1424a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:49:31.160737 25111 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08191518230b4f81a094e80efd1424a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:49:31.160789 25111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08191518230b4f81a094e80efd1424a8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:31.161652 25111 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08191518230b4f81a094e80efd1424a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08191518230b4f81a094e80efd1424a8" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 44341 } }
I20260928 22:49:31.161773 25111 leader_election.cc:302] T 00000000000000000000000000000000 P 08191518230b4f81a094e80efd1424a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08191518230b4f81a094e80efd1424a8; no voters:
I20260928 22:49:31.161983 25111 leader_election.cc:288] T 00000000000000000000000000000000 P 08191518230b4f81a094e80efd1424a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:49:31.162288 25111 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08191518230b4f81a094e80efd1424a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:49:31.162194 25117 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08191518230b4f81a094e80efd1424a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:49:31.162497 25117 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08191518230b4f81a094e80efd1424a8 [term 1 LEADER]: Becoming Leader. State: Replica: 08191518230b4f81a094e80efd1424a8, State: Running, Role: LEADER
Sep 28 22:49:31 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:49:31.162678 25117 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08191518230b4f81a094e80efd1424a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08191518230b4f81a094e80efd1424a8" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 44341 } }
I20260928 22:49:31.163486 25116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08191518230b4f81a094e80efd1424a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08191518230b4f81a094e80efd1424a8. Latest consensus state: current_term: 1 leader_uuid: "08191518230b4f81a094e80efd1424a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08191518230b4f81a094e80efd1424a8" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 44341 } } }
I20260928 22:49:31.163602 25116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08191518230b4f81a094e80efd1424a8 [sys.catalog]: This master's current role is: LEADER
I20260928 22:49:31.163476 25118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08191518230b4f81a094e80efd1424a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08191518230b4f81a094e80efd1424a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08191518230b4f81a094e80efd1424a8" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 44341 } } }
I20260928 22:49:31.164057 25118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08191518230b4f81a094e80efd1424a8 [sys.catalog]: This master's current role is: LEADER
W20260928 22:49:31.164566 25130 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 08191518230b4f81a094e80efd1424a8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:49:31.164623 25130 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:49:31.164675 25131 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:49:31.164868 25131 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:49:31.166518 25131 catalog_manager.cc:1347] Generated new cluster ID: 572f9fd0b3e34579be31bfe14f143ba9
I20260928 22:49:31.166570 25131 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:49:31.167382 25114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:31.155482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42896 (local address 127.16.121.190:44341)
0928 22:49:31.155658 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:31.155666 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:31.155693 (+ 27us) server_negotiation.cc:407] Connection header received
0928 22:49:31.156192 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:31.156200 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:31.156251 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:31.156386 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:31.157267 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:31.158310 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:31.159509 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:31.159988 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:31.160086 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:31.163471 (+ 3385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:31.163477 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:31.163477 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:31.163492 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:31.165587 (+ 2095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:31.166050 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:31.166053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:31.166053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:31.166156 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:31.166363 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:31.166364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:31.166364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:31.166669 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:31.166818 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:31.166984 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 22:49:31.167170 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":83,"threads_started":1}
I20260928 22:49:31.185014 25131 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:49:31.185792 25131 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:49:31.193835 25131 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08191518230b4f81a094e80efd1424a8: Generated new TSK 0
I20260928 22:49:31.194082 25131 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.121.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab.
Entry for principal kudu/127.16.121.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab.
I20260928 22:49:31.232120 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.129:0
--local_ip_for_outbound_sockets=127.16.121.129
--webserver_interface=127.16.121.129
--webserver_port=0
--tserver_master_addrs=127.16.121.190:44341
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab
--principal=kudu/127.16.121.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:41749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:49:31.321749 25137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:31.321993 25137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:31.322021 25137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:31.323374 25137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:31.323429 25137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:31.323462 25137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.129
I20260928 22:49:31.325110 25137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:41749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab
--principal=kudu/127.16.121.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.121.129
--webserver_port=0
--tserver_master_addrs=127.16.121.190:44341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.129
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:31.325403 25137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:31.325676 25137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:31.331238 25145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:31.331379 25143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:31.331538 25142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:31.331715 25137 server_base.cc:1034] running on GCE node
I20260928 22:49:31.331979 25137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:31.332293 25137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:31.333523 25137 hybrid_clock.cc:648] HybridClock initialized: now 1790635771333443 us; error 90 us; skew 500 ppm
Sep 28 22:49:31 dist-test-slave-56vw krb5kdc[25030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635771, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:31.336275 25137 init.cc:377] Logged in from keytab as kudu/127.16.121.129@KRBTEST.COM (short username kudu)
I20260928 22:49:31.336831 25137 webserver.cc:460] Webserver started at http://127.16.121.129:40055/ using document root <none> and password file <none>
I20260928 22:49:31.337086 25137 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:31.337146 25137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:31.337240 25137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:31.338337 25137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-0/data/instance:
uuid: "3f518960965d45ee8be098397ac40403"
format_stamp: "Formatted at 2026-09-28 22:49:31 on dist-test-slave-56vw"
server_key: "7912344263a332d827a579ea25043301"
server_key_iv: "eae7eabc9bfe15b60c5d21e786496c7f"
server_key_version: "encryptionkey@0"
I20260928 22:49:31.338657 25137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance:
uuid: "3f518960965d45ee8be098397ac40403"
format_stamp: "Formatted at 2026-09-28 22:49:31 on dist-test-slave-56vw"
server_key: "7912344263a332d827a579ea25043301"
server_key_iv: "eae7eabc9bfe15b60c5d21e786496c7f"
server_key_version: "encryptionkey@0"
I20260928 22:49:31.340209 25137 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:49:31.341022 25152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:31.341241 25137 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:31.341316 25137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-0/wal
uuid: "3f518960965d45ee8be098397ac40403"
format_stamp: "Formatted at 2026-09-28 22:49:31 on dist-test-slave-56vw"
server_key: "7912344263a332d827a579ea25043301"
server_key_iv: "eae7eabc9bfe15b60c5d21e786496c7f"
server_key_version: "encryptionkey@0"
I20260928 22:49:31.341421 25137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:31.357126 25137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:31.360179 25137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:31.360463 25137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:31.360751 25137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:49:31.361088 25137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:49:31.361145 25137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:31.361186 25137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:49:31.361214 25137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:31.367913 25137 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.129:40073
I20260928 22:49:31.368077 25265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.129:40073 every 8 connection(s)
I20260928 22:49:31.368353 25137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
Sep 28 22:49:31 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635771, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:49:31.377965 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 25137
I20260928 22:49:31.378103 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance
I20260928 22:49:31.378304 16870 external_mini_cluster.cc:1428] Setting key 53381e68498918f20d8f53c00f2e192b
I20260928 22:49:31.378960 25114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:31.369178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:39741 (local address 127.16.121.190:44341)
0928 22:49:31.369265 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:31.369268 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:31.369415 (+ 147us) server_negotiation.cc:407] Connection header received
0928 22:49:31.369942 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:31.369944 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:31.369955 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:31.369999 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:31.370989 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:31.371578 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:31.372670 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:31.372904 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:31.372973 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:31.375652 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:31.375653 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:31.375654 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:31.375661 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:31.377139 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:31.377592 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:31.377593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:31.377593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:31.377650 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:31.378127 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:31.378129 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:31.378129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:31.378312 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:31.378427 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:31.378751 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 22:49:31.378857 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 22:49:31.379139 25268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:31.369099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.129:39741)
0928 22:49:31.369276 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:31.369317 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:31.369829 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:31.370073 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:31.370077 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:31.370680 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:31.370885 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:31.370889 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:31.371671 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:31.371672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:31.372553 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:31.372555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:31.373010 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:31.373011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:31.373134 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:31.373751 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:31.373760 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:31.375490 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:31.377255 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:31.377263 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:31.377265 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:31.377513 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:31.377830 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:31.377831 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:31.377832 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:31.377987 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:31.378463 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:31.378470 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:31.378626 (+ 156us) client_negotiation.cc:769] Sending connection context
0928 22:49:31.378724 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 22:49:31.378859 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":75,"threads_started":1}
I20260928 22:49:31.379716 25266 heartbeater.cc:347] Connected to a master server at 127.16.121.190:44341
I20260928 22:49:31.379835 25266 heartbeater.cc:464] Registering TS with master...
I20260928 22:49:31.380048 25266 heartbeater.cc:511] Master 127.16.121.190:44341 requested a full tablet report, sending...
I20260928 22:49:31.380790 25075 ts_manager.cc:194] Registered new tserver with Master: 3f518960965d45ee8be098397ac40403 (127.16.121.129:40073)
I20260928 22:49:31.381469 25075 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.129@KRBTEST.COM'} at 127.16.121.129:39741
WARNING: no policy specified for kudu/127.16.121.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab.
Entry for principal kudu/127.16.121.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab.
I20260928 22:49:31.416827 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.130:0
--local_ip_for_outbound_sockets=127.16.121.130
--webserver_interface=127.16.121.130
--webserver_port=0
--tserver_master_addrs=127.16.121.190:44341
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab
--principal=kudu/127.16.121.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:41749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:49:31.502873 25271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:31.503118 25271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:31.503142 25271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:31.505005 25271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:31.505065 25271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:31.505096 25271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.130
I20260928 22:49:31.506788 25271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:41749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab
--principal=kudu/127.16.121.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.121.130
--webserver_port=0
--tserver_master_addrs=127.16.121.190:44341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.130
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:31.507107 25271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:31.507364 25271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:31.512948 25276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:31.512991 25277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:31.513121 25271 server_base.cc:1034] running on GCE node
W20260928 22:49:31.512971 25279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:31.513553 25271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:31.513841 25271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:31.515025 25271 hybrid_clock.cc:648] HybridClock initialized: now 1790635771515001 us; error 37 us; skew 500 ppm
Sep 28 22:49:31 dist-test-slave-56vw krb5kdc[25030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635771, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:31.517870 25271 init.cc:377] Logged in from keytab as kudu/127.16.121.130@KRBTEST.COM (short username kudu)
I20260928 22:49:31.518589 25271 webserver.cc:460] Webserver started at http://127.16.121.130:41945/ using document root <none> and password file <none>
I20260928 22:49:31.518867 25271 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:31.518921 25271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:31.519021 25271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:31.520197 25271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-1/data/instance:
uuid: "d5e93f4c601a41be874a7d4681001416"
format_stamp: "Formatted at 2026-09-28 22:49:31 on dist-test-slave-56vw"
server_key: "14780b54890dc9a62e75d62933fedac5"
server_key_iv: "6e0d7a49554aada74e44749b87db7b31"
server_key_version: "encryptionkey@0"
I20260928 22:49:31.520581 25271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance:
uuid: "d5e93f4c601a41be874a7d4681001416"
format_stamp: "Formatted at 2026-09-28 22:49:31 on dist-test-slave-56vw"
server_key: "14780b54890dc9a62e75d62933fedac5"
server_key_iv: "6e0d7a49554aada74e44749b87db7b31"
server_key_version: "encryptionkey@0"
I20260928 22:49:31.522099 25271 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:49:31.523135 25286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:31.523419 25271 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:49:31.523550 25271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-1/wal
uuid: "d5e93f4c601a41be874a7d4681001416"
format_stamp: "Formatted at 2026-09-28 22:49:31 on dist-test-slave-56vw"
server_key: "14780b54890dc9a62e75d62933fedac5"
server_key_iv: "6e0d7a49554aada74e44749b87db7b31"
server_key_version: "encryptionkey@0"
I20260928 22:49:31.523643 25271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:31.549276 25271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:31.552292 25271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:31.552591 25271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:31.552874 25271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:49:31.553256 25271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:49:31.553315 25271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:31.553350 25271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:49:31.553377 25271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:31.560354 25271 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.130:37827
I20260928 22:49:31.560511 25399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.130:37827 every 8 connection(s)
I20260928 22:49:31.560827 25271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
I20260928 22:49:31.563350 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 25271
I20260928 22:49:31.563493 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance
I20260928 22:49:31.563745 16870 external_mini_cluster.cc:1428] Setting key 3e52217ea327e38c045ffc0319d4f0ef
Sep 28 22:49:31 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635771, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.130@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.121.131@KRBTEST.COM; defaulting to no policy
I20260928 22:49:31.576074 25114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:31.561846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:47693 (local address 127.16.121.190:44341)
0928 22:49:31.561915 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:31.561917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:31.562148 (+ 231us) server_negotiation.cc:407] Connection header received
0928 22:49:31.563148 (+ 1000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:31.563150 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:31.563165 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:31.563241 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:31.565034 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:31.565562 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:31.567710 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:31.568049 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:31.568144 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:31.571243 (+ 3099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:31.571245 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:31.571246 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:31.571256 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:31.573245 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:31.574158 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:31.574160 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:31.574160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:31.574239 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:31.574897 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:31.574898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:31.574898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:31.575033 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:31.575188 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:31.575906 (+ 718us) server_negotiation.cc:299] Negotiation successful
0928 22:49:31.576000 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 22:49:31.576269 25402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:31.561841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.130:47693)
0928 22:49:31.562035 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:31.562076 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:31.562993 (+ 917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:31.563350 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:31.563355 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:31.564543 (+ 1188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:31.564825 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:31.564834 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:31.565680 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:31.565682 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:31.567592 (+ 1910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:31.567595 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:31.568198 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:31.568199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:31.568337 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:31.569058 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:31.569068 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:31.571102 (+ 2034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:31.573404 (+ 2302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:31.573422 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:31.573425 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:31.574032 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:31.574338 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:31.574340 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:31.574340 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:31.574633 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:31.575185 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:31.575204 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:31.575596 (+ 392us) client_negotiation.cc:769] Sending connection context
0928 22:49:31.575736 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 22:49:31.575914 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":125,"threads_started":1}
I20260928 22:49:31.577107 25400 heartbeater.cc:347] Connected to a master server at 127.16.121.190:44341
I20260928 22:49:31.577229 25400 heartbeater.cc:464] Registering TS with master...
I20260928 22:49:31.577417 25400 heartbeater.cc:511] Master 127.16.121.190:44341 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.131@KRBTEST.COM" created.
I20260928 22:49:31.578227 25075 ts_manager.cc:194] Registered new tserver with Master: d5e93f4c601a41be874a7d4681001416 (127.16.121.130:37827)
I20260928 22:49:31.579046 25075 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.130@KRBTEST.COM'} at 127.16.121.130:47693
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab.
Entry for principal kudu/127.16.121.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab.
I20260928 22:49:31.611613 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.131:0
--local_ip_for_outbound_sockets=127.16.121.131
--webserver_interface=127.16.121.131
--webserver_port=0
--tserver_master_addrs=127.16.121.190:44341
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab
--principal=kudu/127.16.121.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:41749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:49:31.709005 25405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:31.709282 25405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:31.709331 25405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:31.710758 25405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:31.710855 25405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:31.710906 25405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.131
I20260928 22:49:31.712936 25405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:41749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab
--principal=kudu/127.16.121.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.121.131
--webserver_port=0
--tserver_master_addrs=127.16.121.190:44341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.131
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:31.713469 25405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:31.713949 25405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:31.721365 25410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:31.721323 25411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:31.721407 25405 server_base.cc:1034] running on GCE node
W20260928 22:49:31.721365 25413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:31.721868 25405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:31.722190 25405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:31.723387 25405 hybrid_clock.cc:648] HybridClock initialized: now 1790635771723359 us; error 61 us; skew 500 ppm
Sep 28 22:49:31 dist-test-slave-56vw krb5kdc[25030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635771, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:31.726663 25405 init.cc:377] Logged in from keytab as kudu/127.16.121.131@KRBTEST.COM (short username kudu)
I20260928 22:49:31.727329 25405 webserver.cc:460] Webserver started at http://127.16.121.131:41333/ using document root <none> and password file <none>
I20260928 22:49:31.727646 25405 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:31.727705 25405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:31.727818 25405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:31.728928 25405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-2/data/instance:
uuid: "81e43f77297c4af989539c0a14c6adb4"
format_stamp: "Formatted at 2026-09-28 22:49:31 on dist-test-slave-56vw"
server_key: "5adcc863437a000094c924f22beefeda"
server_key_iv: "cef23f5203cf5442289c53d206ccd1fd"
server_key_version: "encryptionkey@0"
I20260928 22:49:31.729235 25405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance:
uuid: "81e43f77297c4af989539c0a14c6adb4"
format_stamp: "Formatted at 2026-09-28 22:49:31 on dist-test-slave-56vw"
server_key: "5adcc863437a000094c924f22beefeda"
server_key_iv: "cef23f5203cf5442289c53d206ccd1fd"
server_key_version: "encryptionkey@0"
I20260928 22:49:31.730633 25405 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:49:31.731951 25420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:31.732266 25405 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 22:49:31.732456 25405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-2/wal
uuid: "81e43f77297c4af989539c0a14c6adb4"
format_stamp: "Formatted at 2026-09-28 22:49:31 on dist-test-slave-56vw"
server_key: "5adcc863437a000094c924f22beefeda"
server_key_iv: "cef23f5203cf5442289c53d206ccd1fd"
server_key_version: "encryptionkey@0"
I20260928 22:49:31.732554 25405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:31.750100 25405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:31.752782 25405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:31.752985 25405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:31.753268 25405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:49:31.753625 25405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:49:31.753681 25405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:31.753723 25405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:49:31.753752 25405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:31.761332 25405 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.131:44845
I20260928 22:49:31.761421 25533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.131:44845 every 8 connection(s)
I20260928 22:49:31.761826 25405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
I20260928 22:49:31.768214 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 25405
I20260928 22:49:31.768344 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance
I20260928 22:49:31.768656 16870 external_mini_cluster.cc:1428] Setting key 70f6e24969502a2abee30ed801c4d4f0
Sep 28 22:49:31 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635771, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.131@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:49:31.774811 25114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:31.762827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:59337 (local address 127.16.121.190:44341)
0928 22:49:31.762902 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:31.762904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:31.763205 (+ 301us) server_negotiation.cc:407] Connection header received
0928 22:49:31.763887 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:31.763889 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:31.763900 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:31.764127 (+ 227us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:31.765333 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:31.766336 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:31.767897 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:31.768254 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:31.768350 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:31.771695 (+ 3345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:31.771696 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:31.771696 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:31.771706 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:31.773190 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:31.773686 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:31.773688 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:31.773688 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:31.773763 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:31.774063 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:31.774064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:31.774064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:31.774201 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:31.774291 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:31.774613 (+ 322us) server_negotiation.cc:299] Negotiation successful
0928 22:49:31.774705 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 22:49:31.775118 25536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:31.762792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.131:59337)
0928 22:49:31.763022 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:31.763099 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:31.763769 (+ 670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:31.764232 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:31.764236 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:31.764980 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:31.765205 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:31.765210 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:31.766459 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:31.766460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:31.767778 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:31.767781 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:31.768483 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:31.768484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:31.768680 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:31.769381 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:31.769393 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:31.771520 (+ 2127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:31.773314 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:31.773322 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:31.773325 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:31.773591 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:31.773842 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:31.773843 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:31.773843 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:31.773980 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:31.774285 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:31.774293 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:31.774460 (+ 167us) client_negotiation.cc:769] Sending connection context
0928 22:49:31.774577 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 22:49:31.774768 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":113,"threads_started":1}
I20260928 22:49:31.775673 25534 heartbeater.cc:347] Connected to a master server at 127.16.121.190:44341
I20260928 22:49:31.775815 25534 heartbeater.cc:464] Registering TS with master...
I20260928 22:49:31.776015 25534 heartbeater.cc:511] Master 127.16.121.190:44341 requested a full tablet report, sending...
I20260928 22:49:31.776446 25075 ts_manager.cc:194] Registered new tserver with Master: 81e43f77297c4af989539c0a14c6adb4 (127.16.121.131:44845)
I20260928 22:49:31.776882 25075 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.131@KRBTEST.COM'} at 127.16.121.131:59337
I20260928 22:49:31.780613 16870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.16.121.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.121.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.121.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.129.keytab.
Entry for principal oryx/127.16.121.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.129.keytab.
WARNING: no policy specified for oryx/127.16.121.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.121.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.121.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.130.keytab.
Entry for principal oryx/127.16.121.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.130.keytab.
WARNING: no policy specified for oryx/127.16.121.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.121.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.121.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.131.keytab.
Entry for principal oryx/127.16.121.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.131.keytab.
I20260928 22:49:31.881438 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 25137
I20260928 22:49:31.887970 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 25271
I20260928 22:49:31.893622 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 25405
I20260928 22:49:31.900240 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 25043
I20260928 22:49:31.906857 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:44341
--webserver_interface=127.16.121.190
--webserver_port=43461
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:41749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:44341
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:49:32.012578 25543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:32.012961 25543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:32.013007 25543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:32.014472 25543 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:49:32.014567 25543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:32.014590 25543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:32.014611 25543 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:49:32.014640 25543 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:49:32.017679 25543 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:41749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:44341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:44341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.121.190
--webserver_port=43461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:32.018222 25543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:32.018930 25543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:32.025694 25549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:32.025702 25551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:32.026136 25543 server_base.cc:1034] running on GCE node
W20260928 22:49:32.026329 25548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:32.026724 25543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:32.027035 25543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:32.028283 25543 hybrid_clock.cc:648] HybridClock initialized: now 1790635772028266 us; error 32 us; skew 500 ppm
Sep 28 22:49:32 dist-test-slave-56vw krb5kdc[25030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635772, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:32.032258 25543 init.cc:377] Logged in from keytab as kudu/127.16.121.190@KRBTEST.COM (short username kudu)
I20260928 22:49:32.032991 25543 webserver.cc:460] Webserver started at http://127.16.121.190:43461/ using document root <none> and password file <none>
I20260928 22:49:32.033290 25543 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:32.033447 25543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:32.035233 25543 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 22:49:32.036998 25558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:32.037498 25543 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20260928 22:49:32.037606 25543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "08191518230b4f81a094e80efd1424a8"
format_stamp: "Formatted at 2026-09-28 22:49:31 on dist-test-slave-56vw"
server_key: "c9908cae4141884e824ce07982602d26"
server_key_iv: "c6b228d217abdb3b7bf9fcb40d40dcc4"
server_key_version: "encryptionkey@0"
I20260928 22:49:32.038025 25543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:32.062650 25543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:32.065194 25543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:32.065313 25543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:32.069257 25543 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.190:44341
I20260928 22:49:32.069345 25610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.190:44341 every 8 connection(s)
I20260928 22:49:32.069749 25543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:49:32.072124 25611 sys_catalog.cc:263] Verifying existing consensus state
I20260928 22:49:32.072944 25611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08191518230b4f81a094e80efd1424a8: Bootstrap starting.
I20260928 22:49:32.074606 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 25543
I20260928 22:49:32.074944 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.129:40073
--local_ip_for_outbound_sockets=127.16.121.129
--tserver_master_addrs=127.16.121.190:44341
--webserver_port=40055
--webserver_interface=127.16.121.129
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab
--principal=kudu/127.16.121.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:41749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.121.129
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:49:32.083388 25611 log.cc:824] T 00000000000000000000000000000000 P 08191518230b4f81a094e80efd1424a8: Log is configured to *not* fsync() on all Append() calls
I20260928 22:49:32.085541 25611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08191518230b4f81a094e80efd1424a8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 22:49:32.085975 25611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08191518230b4f81a094e80efd1424a8: Bootstrap complete.
I20260928 22:49:32.089529 25611 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08191518230b4f81a094e80efd1424a8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08191518230b4f81a094e80efd1424a8" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 44341 } }
I20260928 22:49:32.089828 25611 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08191518230b4f81a094e80efd1424a8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08191518230b4f81a094e80efd1424a8, State: Initialized, Role: FOLLOWER
I20260928 22:49:32.090035 25611 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08191518230b4f81a094e80efd1424a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08191518230b4f81a094e80efd1424a8" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 44341 } }
I20260928 22:49:32.090117 25611 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08191518230b4f81a094e80efd1424a8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:49:32.090204 25611 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08191518230b4f81a094e80efd1424a8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:49:32.090374 25611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08191518230b4f81a094e80efd1424a8 [term 1 FOLLOWER]: Advancing to term 2
I20260928 22:49:32.092383 25611 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08191518230b4f81a094e80efd1424a8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08191518230b4f81a094e80efd1424a8" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 44341 } }
I20260928 22:49:32.092520 25611 leader_election.cc:302] T 00000000000000000000000000000000 P 08191518230b4f81a094e80efd1424a8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08191518230b4f81a094e80efd1424a8; no voters:
I20260928 22:49:32.092868 25611 leader_election.cc:288] T 00000000000000000000000000000000 P 08191518230b4f81a094e80efd1424a8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 22:49:32.092906 25615 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08191518230b4f81a094e80efd1424a8 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 22:49:32.093045 25615 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08191518230b4f81a094e80efd1424a8 [term 2 LEADER]: Becoming Leader. State: Replica: 08191518230b4f81a094e80efd1424a8, State: Running, Role: LEADER
I20260928 22:49:32.093139 25615 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08191518230b4f81a094e80efd1424a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08191518230b4f81a094e80efd1424a8" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 44341 } }
I20260928 22:49:32.093211 25611 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08191518230b4f81a094e80efd1424a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:49:32.093564 25616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08191518230b4f81a094e80efd1424a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "08191518230b4f81a094e80efd1424a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08191518230b4f81a094e80efd1424a8" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 44341 } } }
I20260928 22:49:32.093825 25616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08191518230b4f81a094e80efd1424a8 [sys.catalog]: This master's current role is: LEADER
I20260928 22:49:32.093758 25617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08191518230b4f81a094e80efd1424a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08191518230b4f81a094e80efd1424a8. Latest consensus state: current_term: 2 leader_uuid: "08191518230b4f81a094e80efd1424a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08191518230b4f81a094e80efd1424a8" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 44341 } } }
I20260928 22:49:32.094233 25617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08191518230b4f81a094e80efd1424a8 [sys.catalog]: This master's current role is: LEADER
I20260928 22:49:32.094497 25627 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:49:32.095141 25627 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:49:32.095504 25627 catalog_manager.cc:1259] Loaded cluster ID: 572f9fd0b3e34579be31bfe14f143ba9
I20260928 22:49:32.095575 25627 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:49:32.096547 25627 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:49:32.096956 25627 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 08191518230b4f81a094e80efd1424a8: Loaded TSK: 0
I20260928 22:49:32.097239 25627 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 22:49:32.192431 25613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:32.192731 25613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:32.192781 25613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:32.194666 25613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:32.194751 25613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:32.194810 25613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.129
I20260928 22:49:32.196753 25613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:41749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.129.keytab
--principal=oryx/127.16.121.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.129:40073
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.121.129
--webserver_port=40055
--tserver_master_addrs=127.16.121.190:44341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.129
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:32.197108 25613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:32.197697 25613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:32.204846 25637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:32.204988 25613 server_base.cc:1034] running on GCE node
W20260928 22:49:32.204807 25638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:32.205289 25640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:32.205557 25613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:32.205941 25613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:32.207226 25613 hybrid_clock.cc:648] HybridClock initialized: now 1790635772207205 us; error 36 us; skew 500 ppm
Sep 28 22:49:32 dist-test-slave-56vw krb5kdc[25030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635772, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.121.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:32.210374 25613 init.cc:377] Logged in from keytab as oryx/127.16.121.129@KRBTEST.COM (short username oryx)
I20260928 22:49:32.211030 25613 webserver.cc:460] Webserver started at http://127.16.121.129:40055/ using document root <none> and password file <none>
I20260928 22:49:32.211364 25613 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:32.211441 25613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:32.213102 25613 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:49:32.213884 25647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:32.214126 25613 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:32.214262 25613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-0/wal
uuid: "3f518960965d45ee8be098397ac40403"
format_stamp: "Formatted at 2026-09-28 22:49:31 on dist-test-slave-56vw"
server_key: "7912344263a332d827a579ea25043301"
server_key_iv: "eae7eabc9bfe15b60c5d21e786496c7f"
server_key_version: "encryptionkey@0"
I20260928 22:49:32.214722 25613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:32.227567 25613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:32.230438 25613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:32.230683 25613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:32.230979 25613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:49:32.231400 25613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:49:32.231465 25613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:32.231518 25613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:49:32.231549 25613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:32.238014 25613 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.129:40073
I20260928 22:49:32.238176 25760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.129:40073 every 8 connection(s)
I20260928 22:49:32.238564 25613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
I20260928 22:49:32.242141 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 25613
I20260928 22:49:32.242601 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.130:37827
--local_ip_for_outbound_sockets=127.16.121.130
--tserver_master_addrs=127.16.121.190:44341
--webserver_port=41945
--webserver_interface=127.16.121.130
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab
--principal=kudu/127.16.121.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:41749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.121.130
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 22:49:32 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.129@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:32 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.129@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
W20260928 22:49:32.251772 25764 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:32.239634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:60133 (local address 127.16.121.190:44341)
0928 22:49:32.239896 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:32.239905 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:32.240154 (+ 249us) server_negotiation.cc:407] Connection header received
0928 22:49:32.241048 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:32.241054 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:32.241102 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:32.241240 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:32.242999 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:32.244704 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:32.245865 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:32.246594 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:32.246708 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:32.251518 (+ 4810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.129:60133: BlockingRecv error: recv got EOF from 127.16.121.129:60133 (error 108)
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":133,"threads_started":1}
I20260928 22:49:32.252560 25763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:32.239515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.129:60133)
0928 22:49:32.239912 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:32.239967 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:32.240706 (+ 739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:32.241315 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:32.241320 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:32.242263 (+ 943us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:32.242557 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:32.242565 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:32.244860 (+ 2295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:32.244861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:32.245722 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:32.245725 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:32.246825 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:32.246826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:32.246995 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:32.247694 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:32.247706 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:32.252301 (+ 4595us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":86,"threads_started":1}
W20260928 22:49:32.253136 25761 heartbeater.cc:644] Failed to heartbeat to 127.16.121.190:44341 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.121.190:44341: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Sep 28 22:49:32 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.129@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:32 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.129@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:32.260026 25763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:32.253452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.129:43221)
0928 22:49:32.253590 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:32.253602 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:32.253662 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:32.253846 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:32.253848 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:32.254036 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:32.254107 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:32.254109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:32.255264 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:32.255266 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:32.256095 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:32.256099 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:32.256633 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:32.256634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:32.256723 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:32.257304 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:32.257310 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:32.259945 (+ 2635us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117}
W20260928 22:49:32.260421 25764 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:32.253540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:43221 (local address 127.16.121.190:44341)
0928 22:49:32.253602 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:32.253604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:32.253646 (+ 42us) server_negotiation.cc:407] Connection header received
0928 22:49:32.253726 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:32.253727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:32.253743 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:32.253825 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:32.254202 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:32.255152 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:32.256203 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:32.256534 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:32.256609 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:32.260260 (+ 3651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.129:43221: BlockingRecv error: recv got EOF from 127.16.121.129:43221 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 28 22:49:32 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.129@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:32 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.129@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:32.266577 25763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:32.260293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.129:53853)
0928 22:49:32.260684 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:32.260695 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:32.260755 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:32.260942 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:32.260944 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:32.261105 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:32.261160 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:32.261162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:32.262303 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:32.262305 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:32.263142 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:32.263146 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:32.263683 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:32.263684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:32.263741 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:32.264291 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:32.264297 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:32.266490 (+ 2193us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":87,"threads_started":1}
W20260928 22:49:32.266691 25764 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:32.260549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:53853 (local address 127.16.121.190:44341)
0928 22:49:32.260679 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:32.260681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:32.260761 (+ 80us) server_negotiation.cc:407] Connection header received
0928 22:49:32.260815 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:32.260816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:32.260829 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:32.261237 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:32.261272 (+ 35us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:32.262160 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:32.263262 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:32.263589 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:32.263666 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:32.266620 (+ 2954us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.129:53853: BlockingRecv error: recv got EOF from 127.16.121.129:53853 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":84,"threads_started":1}
W20260928 22:49:32.267068 25761 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 22:49:32.344467 25765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:32.344765 25765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:32.344827 25765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:32.346290 25765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:32.346386 25765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:32.346436 25765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.130
I20260928 22:49:32.348160 25765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:41749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.130.keytab
--principal=oryx/127.16.121.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.130:37827
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.121.130
--webserver_port=41945
--tserver_master_addrs=127.16.121.190:44341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.130
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:32.348476 25765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:32.348739 25765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:32.354552 25773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:32.354583 25772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:32.354514 25775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:32.354804 25765 server_base.cc:1034] running on GCE node
I20260928 22:49:32.355131 25765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:32.355472 25765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:32.356634 25765 hybrid_clock.cc:648] HybridClock initialized: now 1790635772356614 us; error 68 us; skew 500 ppm
Sep 28 22:49:32 dist-test-slave-56vw krb5kdc[25030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635772, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.121.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:32.360651 25765 init.cc:377] Logged in from keytab as oryx/127.16.121.130@KRBTEST.COM (short username oryx)
I20260928 22:49:32.361217 25765 webserver.cc:460] Webserver started at http://127.16.121.130:41945/ using document root <none> and password file <none>
I20260928 22:49:32.361531 25765 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:32.361627 25765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:32.363540 25765 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 22:49:32.364409 25782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:32.364693 25765 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:32.364817 25765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-1/wal
uuid: "d5e93f4c601a41be874a7d4681001416"
format_stamp: "Formatted at 2026-09-28 22:49:31 on dist-test-slave-56vw"
server_key: "14780b54890dc9a62e75d62933fedac5"
server_key_iv: "6e0d7a49554aada74e44749b87db7b31"
server_key_version: "encryptionkey@0"
I20260928 22:49:32.365242 25765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:32.382035 25765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:32.385506 25765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:32.385740 25765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:32.386045 25765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:49:32.386418 25765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:49:32.386483 25765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:32.386529 25765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:49:32.386564 25765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:32.394801 25765 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.130:37827
I20260928 22:49:32.394896 25895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.130:37827 every 8 connection(s)
I20260928 22:49:32.395354 25765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
I20260928 22:49:32.400269 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 25765
I20260928 22:49:32.400620 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.131:44845
--local_ip_for_outbound_sockets=127.16.121.131
--tserver_master_addrs=127.16.121.190:44341
--webserver_port=41333
--webserver_interface=127.16.121.131
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab
--principal=kudu/127.16.121.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:41749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.121.131
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 22:49:32 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.130@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:32 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.130@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
W20260928 22:49:32.406569 25764 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:32.396348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:34663 (local address 127.16.121.190:44341)
0928 22:49:32.396513 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:32.396515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:32.396919 (+ 404us) server_negotiation.cc:407] Connection header received
0928 22:49:32.397638 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:32.397640 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:32.397658 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:32.397723 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:32.399097 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:32.399728 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:32.401491 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:32.401689 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:32.401772 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:32.406514 (+ 4742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.130:34663: BlockingRecv error: recv got EOF from 127.16.121.130:34663 (error 108)
Metrics: {"server-negotiator.queue_time_us":132}
I20260928 22:49:32.406674 25898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:32.396493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.130:34663)
0928 22:49:32.396724 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:32.396777 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:32.397500 (+ 723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:32.397817 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:32.397821 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:32.398703 (+ 882us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:32.398944 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:32.398951 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:32.399844 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:32.399845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:32.401303 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:32.401307 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:32.401796 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:32.401796 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:32.401980 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:32.402733 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:32.402742 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:32.406340 (+ 3598us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":96,"threads_started":1}
W20260928 22:49:32.407390 25896 heartbeater.cc:644] Failed to heartbeat to 127.16.121.190:44341 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.121.190:44341: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Sep 28 22:49:32 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.130@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:32 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.130@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:32.413209 25898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:32.407849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.130:43937)
0928 22:49:32.407935 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:32.407945 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:32.407984 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:32.408176 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:32.408176 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:32.408347 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:32.408438 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:32.408440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:32.409130 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:32.409131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:32.409659 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:32.409661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:32.410041 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:32.410042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:32.410117 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:32.410658 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:32.410663 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:32.413114 (+ 2451us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":68}
W20260928 22:49:32.413282 25764 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:32.407971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:43937 (local address 127.16.121.190:44341)
0928 22:49:32.408045 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:32.408048 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:32.408056 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:49:32.408091 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:32.408092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:32.408101 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:32.408171 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:32.408523 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:32.409029 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:32.409754 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:32.409960 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:32.410045 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:32.413213 (+ 3168us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.130:43937: BlockingRecv error: recv got EOF from 127.16.121.130:43937 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 22:49:32 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.130@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:32 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.130@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:32.421015 25898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:32.414537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.130:37783)
0928 22:49:32.414605 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:32.414616 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:32.414656 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:32.415074 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:32.415075 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:32.415248 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:32.415297 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:32.415299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:32.416592 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:32.416594 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:32.417441 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:32.417444 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:32.417877 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:32.417878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:32.417941 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:32.418492 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:32.418497 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:32.420928 (+ 2431us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 22:49:32.421175 25764 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:32.414628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:37783 (local address 127.16.121.190:44341)
0928 22:49:32.414722 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:32.414724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:32.414736 (+ 12us) server_negotiation.cc:407] Connection header received
0928 22:49:32.414944 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:32.414945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:32.414961 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:32.415027 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:32.415460 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:32.416440 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:32.417591 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:32.417785 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:32.417856 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:32.421107 (+ 3251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.130:37783: BlockingRecv error: recv got EOF from 127.16.121.130:37783 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260928 22:49:32.421248 25896 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 22:49:32.502800 25899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:32.503116 25899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:32.503157 25899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:32.504493 25899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:32.504551 25899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:32.504582 25899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.131
I20260928 22:49:32.506811 25899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:41749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/krb5kdc/oryx_127.16.121.131.keytab
--principal=oryx/127.16.121.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.131:44845
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.121.131
--webserver_port=41333
--tserver_master_addrs=127.16.121.190:44341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.131
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:32.507136 25899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:32.507373 25899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:32.513473 25904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:32.513875 25907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:32.513921 25905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:32.513912 25899 server_base.cc:1034] running on GCE node
I20260928 22:49:32.514284 25899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:32.514561 25899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:32.515769 25899 hybrid_clock.cc:648] HybridClock initialized: now 1790635772515676 us; error 101 us; skew 500 ppm
Sep 28 22:49:32 dist-test-slave-56vw krb5kdc[25030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635772, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.121.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:32.519338 25899 init.cc:377] Logged in from keytab as oryx/127.16.121.131@KRBTEST.COM (short username oryx)
I20260928 22:49:32.520016 25899 webserver.cc:460] Webserver started at http://127.16.121.131:41333/ using document root <none> and password file <none>
I20260928 22:49:32.520294 25899 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:32.520375 25899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:32.522195 25899 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 22:49:32.523201 25914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:32.523604 25899 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:32.523737 25899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-2/wal
uuid: "81e43f77297c4af989539c0a14c6adb4"
format_stamp: "Formatted at 2026-09-28 22:49:31 on dist-test-slave-56vw"
server_key: "5adcc863437a000094c924f22beefeda"
server_key_iv: "cef23f5203cf5442289c53d206ccd1fd"
server_key_version: "encryptionkey@0"
I20260928 22:49:32.524195 25899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:32.544848 25899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:32.548081 25899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:32.548305 25899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:32.548750 25899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:49:32.549150 25899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:49:32.549213 25899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:32.549259 25899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:49:32.549288 25899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:32.556779 25899 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.131:44845
I20260928 22:49:32.556993 26027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.131:44845 every 8 connection(s)
I20260928 22:49:32.557284 25899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
I20260928 22:49:32.557533 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 25899
Sep 28 22:49:32 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.131@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:32 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.131@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:32.567865 25764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:32.558022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42900 (local address 127.16.121.190:44341)
0928 22:49:32.558081 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:32.558084 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:32.558232 (+ 148us) server_negotiation.cc:407] Connection header received
0928 22:49:32.558359 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:32.558360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:32.558371 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:32.558470 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:32.559381 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:32.559871 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:32.560857 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:32.561218 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:32.561270 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:32.563082 (+ 1812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:32.563088 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:32.563089 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:32.563135 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:32.565784 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:32.566317 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:32.566322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:32.566322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:32.566484 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:32.566801 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:32.566803 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:32.566803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:32.567249 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:32.567514 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:32.567614 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 22:49:32.567756 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260928 22:49:32.568802 25767 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:32.558165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:49227 (local address 127.16.121.190:44341)
0928 22:49:32.558224 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:32.558226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:32.558425 (+ 199us) server_negotiation.cc:407] Connection header received
0928 22:49:32.559168 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:32.559169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:32.559177 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:32.559230 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:32.560497 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:32.561062 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:32.563130 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:32.563545 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:32.563627 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:32.568724 (+ 5097us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.131:49227: BlockingRecv error: recv got EOF from 127.16.121.131:49227 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 22:49:32.568960 26031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:32.558105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.131:49227)
0928 22:49:32.558283 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:32.558334 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:32.559039 (+ 705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:32.559424 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:32.559428 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:32.560100 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:32.560392 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:32.560399 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:32.561193 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:32.561194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:32.562978 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:32.562982 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:32.563710 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:32.563711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:32.564018 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:32.564916 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:32.564926 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:32.568632 (+ 3706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":85,"threads_started":1}
W20260928 22:49:32.569540 26028 heartbeater.cc:644] Failed to heartbeat to 127.16.121.190:44341 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.121.190:44341: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Sep 28 22:49:32 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.131@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:32 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.131@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:32.577745 26031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:32.569821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.131:59505)
0928 22:49:32.569879 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:32.569891 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:32.569938 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:32.570636 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:32.570638 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:32.570837 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:32.570993 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:32.570997 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:32.571809 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:32.571810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:32.572839 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:32.572843 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:32.573606 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:32.573608 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:32.573697 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:32.574270 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:32.574276 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:32.577599 (+ 3323us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 22:49:32.577811 25767 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:32.570430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:59505 (local address 127.16.121.190:44341)
0928 22:49:32.570473 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:32.570475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:32.570484 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:49:32.570504 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:32.570505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:32.570515 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:32.570603 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:32.571118 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:32.571720 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:32.573114 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:32.573455 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:32.573536 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:32.577727 (+ 4191us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.131:59505: BlockingRecv error: recv got EOF from 127.16.121.131:59505 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Sep 28 22:49:32 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.131@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:32 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.131@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:32.584519 26031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:32.578184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.131:36153)
0928 22:49:32.578264 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:32.578278 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:32.578340 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:32.578501 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:32.578502 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:32.578709 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:32.578763 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:32.578765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:32.579817 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:32.579818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:32.580597 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:32.580601 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:32.581077 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:32.581078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:32.581140 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:32.581809 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:32.581814 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:32.584416 (+ 2602us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 22:49:32.584599 25767 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:32.578294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:36153 (local address 127.16.121.190:44341)
0928 22:49:32.578344 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:32.578347 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:32.578357 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:49:32.578416 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:32.578417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:32.578430 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:32.578494 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:32.578858 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:32.579723 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:32.580735 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:32.580978 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:32.581076 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:32.584523 (+ 3447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.131:36153: BlockingRecv error: recv got EOF from 127.16.121.131:36153 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260928 22:49:32.584757 26028 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 22:49:33 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.129@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:33 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.129@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:33.274044 26032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:33.267684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.129:55477)
0928 22:49:33.267816 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:33.267836 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:33.267903 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:33.268100 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:33.268101 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:33.268267 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:33.268318 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:33.268319 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:33.269342 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:33.269343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:33.269925 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:33.269928 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:33.270358 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:33.270359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:33.270411 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:33.271211 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:33.271218 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:33.273968 (+ 2750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":80,"threads_started":1}
W20260928 22:49:33.274217 26033 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:33.267776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:55477 (local address 127.16.121.190:44341)
0928 22:49:33.267939 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:33.267941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:33.267951 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:49:33.267980 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:33.267981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:33.267991 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:33.268076 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:33.268461 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:33.269213 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:33.270050 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:33.270262 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:33.270332 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:33.274115 (+ 3783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.129:55477: BlockingRecv error: recv got EOF from 127.16.121.129:55477 (error 108)
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":103,"threads_started":1}
Sep 28 22:49:33 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.130@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:33 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.130@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:33.427417 26034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:33.421715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.130:50873)
0928 22:49:33.421885 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:33.421899 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:33.421996 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:33.422387 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:33.422388 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:33.422589 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:33.422645 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:33.422647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:33.423521 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:33.423522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:33.424156 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:33.424159 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:33.424641 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:33.424642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:33.424695 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:33.425225 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:33.425231 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:33.427342 (+ 2111us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":71,"threads_started":1}
W20260928 22:49:33.427644 26033 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:33.421772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:50873 (local address 127.16.121.190:44341)
0928 22:49:33.421823 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:33.421826 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:33.421987 (+ 161us) server_negotiation.cc:407] Connection header received
0928 22:49:33.422073 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:33.422074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:33.422088 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:33.422151 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:33.422785 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:33.423384 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:33.424264 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:33.424524 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:33.424606 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:33.427558 (+ 2952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.130:50873: BlockingRecv error: recv got EOF from 127.16.121.130:50873 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 28 22:49:33 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.131@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:33 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.131@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:33.591037 26035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:33.585202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.131:42255)
0928 22:49:33.585411 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:33.585432 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:33.585506 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:33.585676 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:33.585678 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:33.585876 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:33.585939 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:33.585942 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:33.586681 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:33.586683 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:33.587337 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:33.587340 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:33.587798 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:33.587799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:33.587858 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:33.588609 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:33.588614 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:33.590952 (+ 2338us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":62,"threads_started":1}
W20260928 22:49:33.591347 26033 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:33.585347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:42255 (local address 127.16.121.190:44341)
0928 22:49:33.585411 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:33.585413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:33.585489 (+ 76us) server_negotiation.cc:407] Connection header received
0928 22:49:33.585556 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:33.585557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:33.585570 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:33.585621 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:33.586063 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:33.586559 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:33.587453 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:33.587655 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:33.587722 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:33.591201 (+ 3479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.131:42255: BlockingRecv error: recv got EOF from 127.16.121.131:42255 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 28 22:49:34 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.129@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:34 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.129@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:34.280375 26036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:34.274736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.129:41793)
0928 22:49:34.274990 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:34.275002 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:34.275099 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:34.275379 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:34.275380 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:34.275554 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:34.275604 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:34.275606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:34.276378 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:34.276379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:34.276995 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:34.276998 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:34.277459 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:34.277460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:34.277509 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:34.278096 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:34.278100 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:34.280309 (+ 2209us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":143,"threads_started":1}
W20260928 22:49:34.280563 26037 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:34.274810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:41793 (local address 127.16.121.190:44341)
0928 22:49:34.274985 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:34.274987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:34.275123 (+ 136us) server_negotiation.cc:407] Connection header received
0928 22:49:34.275201 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:34.275202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:34.275250 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:34.275335 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:34.275711 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:34.276289 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:34.277105 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:34.277318 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:34.277384 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:34.280451 (+ 3067us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.129:41793: BlockingRecv error: recv got EOF from 127.16.121.129:41793 (error 108)
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":65,"threads_started":1}
Sep 28 22:49:34 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.130@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:34 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.130@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:34.433060 26038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:34.427920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.130:46807)
0928 22:49:34.428102 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:34.428122 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:34.428188 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:34.428378 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:34.428379 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:34.428616 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:34.428671 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:34.428673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:34.429359 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:34.429360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:34.429977 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:34.429979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:34.430467 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:34.430468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:34.430543 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:34.431134 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:34.431140 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:34.432998 (+ 1858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":68,"threads_started":1}
W20260928 22:49:34.433223 26037 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:34.428117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:46807 (local address 127.16.121.190:44341)
0928 22:49:34.428181 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:34.428183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:34.428194 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:49:34.428260 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:34.428261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:34.428272 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:34.428327 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:34.428767 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:34.429268 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:34.430104 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:34.430357 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:34.430467 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:34.433152 (+ 2685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.130:46807: BlockingRecv error: recv got EOF from 127.16.121.130:46807 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 22:49:34 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.131@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:34 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.131@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:34.597082 26039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:34.591695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.131:39129)
0928 22:49:34.591836 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:34.591850 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:34.591918 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:34.592092 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:34.592093 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:34.592282 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:34.592332 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:34.592335 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:34.593126 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:34.593127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:34.593911 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:34.593914 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:34.594320 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:34.594321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:34.594375 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:34.594974 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:34.594979 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:34.596982 (+ 2003us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":70,"threads_started":1}
W20260928 22:49:34.597137 26037 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:34.591834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:39129 (local address 127.16.121.190:44341)
0928 22:49:34.591914 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:34.591916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:34.591928 (+ 12us) server_negotiation.cc:407] Connection header received
0928 22:49:34.591984 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:34.591985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:34.592000 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:34.592085 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:34.592464 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:34.593019 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:34.594026 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:34.594231 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:34.594321 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:34.597073 (+ 2752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.131:39129: BlockingRecv error: recv got EOF from 127.16.121.131:39129 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 22:49:35 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.129@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:35 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.129@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:35.286677 26040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:35.281065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.129:42367)
0928 22:49:35.281263 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:35.281275 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:35.281348 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:35.281607 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:35.281609 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:35.281780 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:35.281833 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:35.281835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:35.282671 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:35.282672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:35.283435 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:35.283439 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:35.283827 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:35.283828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:35.283871 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:35.284416 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:35.284421 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:35.286613 (+ 2192us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":111,"threads_started":1}
W20260928 22:49:35.286931 26041 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:35.281187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:42367 (local address 127.16.121.190:44341)
0928 22:49:35.281318 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:35.281320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:35.281330 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:49:35.281482 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:35.281483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:35.281494 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:35.281560 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:35.281958 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:35.282580 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:35.283542 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:35.283755 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:35.283816 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:35.286814 (+ 2998us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.129:42367: BlockingRecv error: recv got EOF from 127.16.121.129:42367 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":74,"threads_started":1}
Sep 28 22:49:35 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.130@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:35 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.130@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:35.438781 26042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:35.433695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.130:59071)
0928 22:49:35.433857 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:35.433873 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:35.433950 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:35.434139 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:35.434141 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:35.434331 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:35.434382 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:35.434384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:35.435171 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:35.435173 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:35.435923 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:35.435926 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:35.436288 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:35.436289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:35.436343 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:35.436854 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:35.436859 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:35.438720 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":91,"threads_started":1}
W20260928 22:49:35.438899 26041 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:35.433819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:59071 (local address 127.16.121.190:44341)
0928 22:49:35.433896 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:35.433899 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:35.433930 (+ 31us) server_negotiation.cc:407] Connection header received
0928 22:49:35.434040 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:35.434041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:35.434061 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:35.434128 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:35.434472 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:35.435089 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:35.436014 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:35.436215 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:35.436291 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:35.438833 (+ 2542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.130:59071: BlockingRecv error: recv got EOF from 127.16.121.130:59071 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 22:49:35 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.131@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:35 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.131@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:35.603587 26043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:35.597738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.131:53615)
0928 22:49:35.597893 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:35.597906 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:35.597974 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:35.598118 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:35.598118 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:35.598288 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:35.598349 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:35.598351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:35.599114 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:35.599115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:35.599785 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:35.599788 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:35.600179 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:35.600180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:35.600228 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:35.601052 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:35.601057 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:35.603519 (+ 2462us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":95,"threads_started":1}
W20260928 22:49:35.603683 26041 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:35.597836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:53615 (local address 127.16.121.190:44341)
0928 22:49:35.597899 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:35.597901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:35.597961 (+ 60us) server_negotiation.cc:407] Connection header received
0928 22:49:35.598036 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:35.598036 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:35.598048 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:35.598093 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:35.598462 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:35.598975 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:35.599902 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:35.600094 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:35.600167 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:35.603608 (+ 3441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.131:53615: BlockingRecv error: recv got EOF from 127.16.121.131:53615 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 28 22:49:36 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.129@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:36 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.129@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:36.297037 26044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:36.287320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.129:40729)
0928 22:49:36.287527 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:36.287539 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:36.287601 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:36.287790 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:36.287792 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:36.287979 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:36.288033 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:36.288035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:36.288757 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:36.288758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:36.289426 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:36.289429 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:36.289833 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:36.289835 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:36.289884 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:36.294621 (+ 4737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:36.294628 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:36.296954 (+ 2326us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":101,"threads_started":1}
W20260928 22:49:36.297252 26045 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:36.287446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:40729 (local address 127.16.121.190:44341)
0928 22:49:36.287593 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:36.287596 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:36.287605 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:49:36.287653 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:36.287653 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:36.287664 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:36.287725 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:36.288120 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:36.288663 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:36.289526 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:36.289740 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:36.289809 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:36.297170 (+ 7361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.129:40729: BlockingRecv error: recv got EOF from 127.16.121.129:40729 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":84,"threads_started":1}
Sep 28 22:49:36 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.130@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:36 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.130@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:36.444870 26046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:36.439581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.130:43941)
0928 22:49:36.439729 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:36.439743 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:36.439815 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:36.440066 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:36.440067 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:36.440255 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:36.440303 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:36.440304 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:36.440979 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:36.440980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:36.441518 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:36.441519 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:36.441905 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:36.441905 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:36.441968 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:36.442514 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:36.442519 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:36.444811 (+ 2292us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":97,"threads_started":1}
W20260928 22:49:36.445014 26045 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:36.439706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:43941 (local address 127.16.121.190:44341)
0928 22:49:36.439793 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:36.439795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:36.439808 (+ 13us) server_negotiation.cc:407] Connection header received
0928 22:49:36.439967 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:36.439968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:36.439989 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:36.440057 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:36.440392 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:36.440893 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:36.441604 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:36.441803 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:36.441886 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:36.444925 (+ 3039us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.130:43941: BlockingRecv error: recv got EOF from 127.16.121.130:43941 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 28 22:49:36 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.131@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:36 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.131@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:36.609548 26047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:36.604251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.131:45435)
0928 22:49:36.604500 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:36.604520 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:36.604582 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:36.604786 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:36.604788 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:36.604963 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:36.605022 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:36.605024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:36.605750 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:36.605752 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:36.606278 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:36.606280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:36.606771 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:36.606772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:36.606842 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:36.607336 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:36.607341 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:36.609447 (+ 2106us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":192,"threads_started":1}
W20260928 22:49:36.609629 26045 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:36.604339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:45435 (local address 127.16.121.190:44341)
0928 22:49:36.604417 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:36.604420 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:36.604588 (+ 168us) server_negotiation.cc:407] Connection header received
0928 22:49:36.604652 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:36.604653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:36.604668 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:36.604716 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:36.605124 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:36.605662 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:36.606471 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:36.606663 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:36.606737 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:36.609557 (+ 2820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.131:45435: BlockingRecv error: recv got EOF from 127.16.121.131:45435 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 22:49:37 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.129@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:37 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.129@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:37.303308 26048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:37.297627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.129:41541)
0928 22:49:37.297793 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:37.297806 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:37.297872 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:37.298088 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:37.298089 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:37.298346 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:37.298397 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:37.298399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:37.299172 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:37.299177 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:37.299823 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:37.299826 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:37.300325 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:37.300326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:37.300373 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:37.300980 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:37.300985 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:37.303172 (+ 2187us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":105,"threads_started":1}
W20260928 22:49:37.303355 26049 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:37.297788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:41541 (local address 127.16.121.190:44341)
0928 22:49:37.297915 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:37.297917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:37.297926 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:49:37.297953 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:37.297953 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:37.297963 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:37.298043 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:37.298514 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:37.299034 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:37.299938 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:37.300183 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:37.300266 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:37.303293 (+ 3027us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.129:41541: BlockingRecv error: recv got EOF from 127.16.121.129:41541 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":72,"threads_started":1}
Sep 28 22:49:37 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.130@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:37 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.130@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:37.451588 26050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:37.445549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.130:38645)
0928 22:49:37.445732 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:37.445746 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:37.445819 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:37.446042 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:37.446043 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:37.446225 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:37.446274 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:37.446276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:37.446963 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:37.446965 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:37.447528 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:37.447530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:37.447938 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:37.447939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:37.448042 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:37.448793 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:37.448798 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:37.451508 (+ 2710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":95,"threads_started":1}
W20260928 22:49:37.451696 26049 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:37.445686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:38645 (local address 127.16.121.190:44341)
0928 22:49:37.445760 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:37.445763 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:37.445800 (+ 37us) server_negotiation.cc:407] Connection header received
0928 22:49:37.445944 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:37.445945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:37.445959 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:37.446033 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:37.446369 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:37.446872 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:37.447617 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:37.447836 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:37.447922 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:37.451646 (+ 3724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.130:38645: BlockingRecv error: recv got EOF from 127.16.121.130:38645 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 22:49:37 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.131@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:37 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.131@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:37.616200 26051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:37.610182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.131:46429)
0928 22:49:37.610409 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:37.610421 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:37.610496 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:37.610679 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:37.610680 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:37.610882 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:37.610933 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:37.610934 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:37.611899 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:37.611901 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:37.612518 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:37.612521 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:37.613006 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:37.613007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:37.613059 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:37.613811 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:37.613815 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:37.616136 (+ 2321us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":90,"threads_started":1}
W20260928 22:49:37.616567 26049 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:37.610302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:46429 (local address 127.16.121.190:44341)
0928 22:49:37.610382 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:37.610384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:37.610543 (+ 159us) server_negotiation.cc:407] Connection header received
0928 22:49:37.610575 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:37.610576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:37.610591 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:37.610638 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:37.611033 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:37.611806 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:37.612627 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:37.612841 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:37.612923 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:37.616496 (+ 3573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.131:46429: BlockingRecv error: recv got EOF from 127.16.121.131:46429 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 22:49:38 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.129@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:38 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.129@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:38.309077 26052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:38.304009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.129:50079)
0928 22:49:38.304189 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:38.304200 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:38.304252 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:38.304400 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:38.304401 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:38.304563 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:38.304614 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:38.304615 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:38.305329 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:38.305330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:38.305898 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:38.305900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:38.306440 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:38.306441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:38.306520 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:38.307209 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:38.307216 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:38.309016 (+ 1800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":104,"threads_started":1}
W20260928 22:49:38.309260 26053 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:38.304120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:50079 (local address 127.16.121.190:44341)
0928 22:49:38.304226 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:38.304228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:38.304240 (+ 12us) server_negotiation.cc:407] Connection header received
0928 22:49:38.304310 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:38.304310 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:38.304321 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:38.304370 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:38.304698 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:38.305237 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:38.305995 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:38.306262 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:38.306348 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:38.309139 (+ 2791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.129:50079: BlockingRecv error: recv got EOF from 127.16.121.129:50079 (error 108)
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":51,"threads_started":1}
Sep 28 22:49:38 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.130@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:38 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.130@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:38.457417 26054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:38.452241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.130:37349)
0928 22:49:38.452435 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:38.452450 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:38.452520 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:38.452667 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:38.452667 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:38.452838 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:38.452886 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:38.452887 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:38.453620 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:38.453621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:38.454180 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:38.454182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:38.454670 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:38.454671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:38.454725 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:38.455378 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:38.455383 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:38.457354 (+ 1971us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":76,"threads_started":1}
W20260928 22:49:38.457687 26053 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:38.452425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:37349 (local address 127.16.121.190:44341)
0928 22:49:38.452513 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:38.452515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:38.452526 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:49:38.452569 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:38.452570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:38.452582 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:38.452640 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:38.452996 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:38.453518 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:38.454378 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:38.454575 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:38.454641 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:38.457595 (+ 2954us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.130:37349: BlockingRecv error: recv got EOF from 127.16.121.130:37349 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 22:49:38 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.131@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:38 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.131@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:38.622323 26055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:38.616827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.131:57259)
0928 22:49:38.617013 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:38.617025 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:38.617073 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:38.617240 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:38.617241 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:38.617400 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:38.617450 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:38.617451 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:38.618458 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:38.618460 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:38.619217 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:38.619220 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:38.619600 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:38.619601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:38.619653 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:38.620236 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:38.620241 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:38.622261 (+ 2020us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":98,"threads_started":1}
W20260928 22:49:38.622510 26053 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:38.616962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:57259 (local address 127.16.121.190:44341)
0928 22:49:38.617040 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:38.617042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:38.617087 (+ 45us) server_negotiation.cc:407] Connection header received
0928 22:49:38.617149 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:38.617149 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:38.617161 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:38.617214 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:38.617581 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:38.618378 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:38.619310 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:38.619506 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:38.619567 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:38.622413 (+ 2846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.131:57259: BlockingRecv error: recv got EOF from 127.16.121.131:57259 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 22:49:39 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.129@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:39 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.129@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:39.314637 26056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:39.309640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.129:52371)
0928 22:49:39.309861 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:39.309873 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:39.309940 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:39.310102 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:39.310103 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:39.310269 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:39.310320 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:39.310321 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:39.311010 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:39.311011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:39.311620 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:39.311622 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:39.312007 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:39.312008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:39.312050 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:39.312630 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:39.312635 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:39.314575 (+ 1940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":140,"threads_started":1}
W20260928 22:49:39.314776 26057 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:39.309752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:52371 (local address 127.16.121.190:44341)
0928 22:49:39.309869 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:39.309871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:39.309944 (+ 73us) server_negotiation.cc:407] Connection header received
0928 22:49:39.310008 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:39.310009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:39.310019 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:39.310071 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:39.310401 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:39.310911 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:39.311715 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:39.311926 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:39.311996 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:39.314683 (+ 2687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.129:52371: BlockingRecv error: recv got EOF from 127.16.121.129:52371 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":58,"threads_started":1}
Sep 28 22:49:39 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.130@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:39 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.130@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:39.463596 26058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:39.458040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.130:54119)
0928 22:49:39.458290 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:39.458306 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:39.458388 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:39.458608 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:39.458609 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:39.458838 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:39.458890 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:39.458893 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:39.459836 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:39.459838 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:39.460413 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:39.460415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:39.460872 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:39.460873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:39.460940 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:39.461418 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:39.461423 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:39.463514 (+ 2091us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":103,"threads_started":1}
W20260928 22:49:39.463725 26057 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:39.458176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:54119 (local address 127.16.121.190:44341)
0928 22:49:39.458385 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:39.458388 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:39.458401 (+ 13us) server_negotiation.cc:407] Connection header received
0928 22:49:39.458509 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:39.458510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:39.458523 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:39.458601 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:39.458989 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:39.459653 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:39.460526 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:39.460750 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:39.460847 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:39.463613 (+ 2766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.130:54119: BlockingRecv error: recv got EOF from 127.16.121.130:54119 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 28 22:49:39 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.131@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:39 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.131@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:39.628451 26059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:39.622895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.131:51369)
0928 22:49:39.623190 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:39.623210 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:39.623316 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:39.623516 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:39.623517 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:39.623848 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:39.623917 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:39.623921 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:39.624665 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:39.624667 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:39.625332 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:39.625335 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:39.625734 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:39.625735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:39.625823 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:39.626380 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:39.626386 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:39.628347 (+ 1961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":114,"threads_started":1}
W20260928 22:49:39.628566 26057 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:39.623073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:51369 (local address 127.16.121.190:44341)
0928 22:49:39.623164 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:39.623167 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:39.623311 (+ 144us) server_negotiation.cc:407] Connection header received
0928 22:49:39.623390 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:39.623391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:39.623407 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:39.623489 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:39.624009 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:39.624571 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:39.625424 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:39.625651 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:39.625710 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:39.628496 (+ 2786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.131:51369: BlockingRecv error: recv got EOF from 127.16.121.131:51369 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 22:49:40 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.129@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:40 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.129@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:40.320876 26060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:40.315187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.129:46963)
0928 22:49:40.315350 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:40.315366 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:40.315430 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:40.315734 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:40.315736 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:40.315921 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:40.315975 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:40.315977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:40.316958 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:40.316959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:40.317548 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:40.317551 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:40.318002 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:40.318003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:40.318049 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:40.318573 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:40.318578 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:40.320796 (+ 2218us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":61,"threads_started":1}
W20260928 22:49:40.321094 26061 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:40.315392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:46963 (local address 127.16.121.190:44341)
0928 22:49:40.315562 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:40.315564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:40.315572 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:49:40.315595 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:40.315595 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:40.315605 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:40.315672 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:40.316113 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:40.316852 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:40.317706 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:40.317922 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:40.317993 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:40.321017 (+ 3024us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.129:46963: BlockingRecv error: recv got EOF from 127.16.121.129:46963 (error 108)
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":96,"threads_started":1}
Sep 28 22:49:40 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.130@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:40 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.130@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:40.470314 26062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:40.464108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.130:34863)
0928 22:49:40.464264 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:40.464275 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:40.464343 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:40.464484 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:40.464485 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:40.464649 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:40.464702 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:40.464703 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:40.465566 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:40.465567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:40.466108 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:40.466110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:40.466822 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:40.466823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:40.466880 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:40.467535 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:40.467540 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:40.470248 (+ 2708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":86,"threads_started":1}
W20260928 22:49:40.470549 26061 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:40.464176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:34863 (local address 127.16.121.190:44341)
0928 22:49:40.464242 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:40.464245 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:40.464345 (+ 100us) server_negotiation.cc:407] Connection header received
0928 22:49:40.464395 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:40.464396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:40.464408 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:40.464459 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:40.464762 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:40.465481 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:40.466241 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:40.466633 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:40.466712 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:40.470467 (+ 3755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.130:34863: BlockingRecv error: recv got EOF from 127.16.121.130:34863 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 28 22:49:40 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.131@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:40 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.131@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:40.635515 26063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:40.629107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.131:43833)
0928 22:49:40.629315 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:40.629330 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:40.629409 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:40.629655 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:40.629656 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:40.629910 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:40.629961 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:40.629963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:40.631109 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:40.631110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:40.632044 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:40.632047 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:40.632586 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:40.632587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:40.632656 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:40.633180 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:40.633185 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:40.635408 (+ 2223us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":100,"threads_started":1}
W20260928 22:49:40.635617 26061 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:40.629289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:43833 (local address 127.16.121.190:44341)
0928 22:49:40.629425 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:40.629427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:40.629472 (+ 45us) server_negotiation.cc:407] Connection header received
0928 22:49:40.629513 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:40.629514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:40.629527 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:40.629589 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:40.630096 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:40.630888 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:40.632165 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:40.632483 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:40.632582 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:40.635545 (+ 2963us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.131:43833: BlockingRecv error: recv got EOF from 127.16.121.131:43833 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Sep 28 22:49:41 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.129@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:41 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.129@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:41.333549 26064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:41.321394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.129:43505)
0928 22:49:41.321563 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:41.321579 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:41.321659 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:41.321947 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:41.321948 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:41.322119 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:41.322171 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:41.322173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:41.323168 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:41.323169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:41.324186 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:41.324191 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:41.324653 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:41.324654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:41.324711 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:41.331409 (+ 6698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:41.331416 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:41.333486 (+ 2070us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":67,"threads_started":1}
W20260928 22:49:41.333765 26065 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:41.321634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:43505 (local address 127.16.121.190:44341)
0928 22:49:41.321805 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:41.321807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:41.321816 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:49:41.321842 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:41.321843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:41.321852 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:41.321917 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:41.322309 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:41.322932 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:41.324311 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:41.324532 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:41.324602 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:41.333682 (+ 9080us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.129:43505: BlockingRecv error: recv got EOF from 127.16.121.129:43505 (error 108)
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":110,"threads_started":1}
Sep 28 22:49:41 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.130@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:41 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.130@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:41.477705 26066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:41.470954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.130:52067)
0928 22:49:41.471118 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:41.471131 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:41.471198 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:41.471414 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:41.471415 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:41.471623 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:41.471675 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:41.471677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:41.472641 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:41.472642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:41.473224 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:41.473228 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:41.473700 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:41.473701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:41.473752 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:41.474373 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:41.474378 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:41.477592 (+ 3214us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":130,"threads_started":1}
W20260928 22:49:41.477761 26065 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:41.471109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:52067 (local address 127.16.121.190:44341)
0928 22:49:41.471191 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:41.471193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:41.471205 (+ 12us) server_negotiation.cc:407] Connection header received
0928 22:49:41.471256 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:41.471257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:41.471271 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:41.471365 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:41.471797 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:41.472387 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:41.473351 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:41.473561 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:41.473630 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:41.477694 (+ 4064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.130:52067: BlockingRecv error: recv got EOF from 127.16.121.130:52067 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 28 22:49:41 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.131@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:41 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.131@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
W20260928 22:49:41.643150 26065 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:41.636502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:45407 (local address 127.16.121.190:44341)
0928 22:49:41.636620 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:41.636622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:41.636930 (+ 308us) server_negotiation.cc:407] Connection header received
0928 22:49:41.636969 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:41.636970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:41.636985 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:41.637074 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:41.637661 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:41.638354 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:41.639335 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:41.639740 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:41.639827 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:41.643056 (+ 3229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.131:45407: BlockingRecv error: recv got EOF from 127.16.121.131:45407 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 22:49:41.643167 26067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:41.636352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.131:45407)
0928 22:49:41.636600 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:41.636611 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:41.636734 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:41.637120 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:41.637122 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:41.637468 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:41.637534 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:41.637537 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:41.638462 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:41.638464 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:41.639015 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:41.639017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:41.639876 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:41.639877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:41.639938 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:41.640540 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:41.640545 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:41.642858 (+ 2313us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":77,"threads_started":1}
Sep 28 22:49:42 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.129@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:42 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.129@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:42.339537 26068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:42.334078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.129:46713)
0928 22:49:42.334227 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:42.334239 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:42.334298 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:42.334436 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:42.334437 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:42.334607 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:42.334657 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:42.334659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:42.335465 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:42.335466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:42.335995 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:42.335997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:42.336419 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:42.336421 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:42.336478 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:42.337120 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:42.337126 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:42.339420 (+ 2294us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":92,"threads_started":1}
W20260928 22:49:42.339589 26069 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:42.334129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:46713 (local address 127.16.121.190:44341)
0928 22:49:42.334272 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:42.334274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:42.334284 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:49:42.334352 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:42.334352 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:42.334363 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:42.334412 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:42.334731 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:42.335333 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:42.336073 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:42.336263 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:42.336322 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:42.339512 (+ 3190us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.129:46713: BlockingRecv error: recv got EOF from 127.16.121.129:46713 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":88,"threads_started":1}
Sep 28 22:49:42 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.130@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:42 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.130@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:42.484093 26070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:42.478391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.130:54717)
0928 22:49:42.478629 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:42.478643 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:42.478755 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:42.478951 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:42.478953 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:42.479253 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:42.479313 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:42.479317 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:42.480086 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:42.480087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:42.480681 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:42.480683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:42.481148 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:42.481149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:42.481213 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:42.481736 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:42.481741 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:42.484031 (+ 2290us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":127,"threads_started":1}
W20260928 22:49:42.484418 26069 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:42.478533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:54717 (local address 127.16.121.190:44341)
0928 22:49:42.478631 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:42.478633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:42.478756 (+ 123us) server_negotiation.cc:407] Connection header received
0928 22:49:42.478828 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:42.478829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:42.478842 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:42.478939 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:42.479419 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:42.479981 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:42.480777 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:42.481030 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:42.481149 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:42.484239 (+ 3090us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.130:54717: BlockingRecv error: recv got EOF from 127.16.121.130:54717 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 28 22:49:42 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.131@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:42 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.131@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:42.649191 26071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:42.643745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.131:44147)
0928 22:49:42.643892 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:42.643903 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:42.643956 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:42.644181 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:42.644182 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:42.644436 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:42.644489 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:42.644492 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:42.645458 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:42.645459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:42.646113 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:42.646115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:42.646543 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:42.646544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:42.646607 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:42.647213 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:42.647225 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:42.649081 (+ 1856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":85,"threads_started":1}
W20260928 22:49:42.649245 26069 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:42.643928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:44147 (local address 127.16.121.190:44341)
0928 22:49:42.644000 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:42.644006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:42.644018 (+ 12us) server_negotiation.cc:407] Connection header received
0928 22:49:42.644046 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:42.644046 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:42.644057 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:42.644122 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:42.644601 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:42.645365 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:42.646207 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:42.646444 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:42.646538 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:42.649162 (+ 2624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.131:44147: BlockingRecv error: recv got EOF from 127.16.121.131:44147 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 28 22:49:43 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.129@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:43 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.129@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:43.345312 26072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:43.340108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.129:43529)
0928 22:49:43.340268 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:43.340279 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:43.340335 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:43.340518 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:43.340518 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:43.340688 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:43.340736 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:43.340738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:43.341464 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:43.341465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:43.342064 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:43.342067 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:43.342662 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:43.342662 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:43.342713 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:43.343267 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:43.343272 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:43.345167 (+ 1895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":95,"threads_started":1}
W20260928 22:49:43.345319 26073 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:43.340212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:43529 (local address 127.16.121.190:44341)
0928 22:49:43.340381 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:43.340383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:43.340392 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:49:43.340417 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:43.340417 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:43.340428 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:43.340499 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:43.340814 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:43.341355 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:43.342224 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:43.342558 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:43.342627 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:43.345263 (+ 2636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.129:43529: BlockingRecv error: recv got EOF from 127.16.121.129:43529 (error 108)
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":72,"threads_started":1}
Sep 28 22:49:43 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.130@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:43 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.130@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:43.496219 26074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:43.484602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.130:33823)
0928 22:49:43.484765 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:43.484792 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:43.484863 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:43.485015 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:43.485016 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:43.485210 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:43.485261 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:43.485263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:43.486012 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:43.486013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:43.486589 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:43.486592 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:43.487556 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:43.487557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:43.487630 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:43.493676 (+ 6046us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:43.493683 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:43.496112 (+ 2429us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":103,"threads_started":1}
W20260928 22:49:43.496295 26073 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:43.484731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:33823 (local address 127.16.121.190:44341)
0928 22:49:43.484794 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:43.484796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:43.484844 (+ 48us) server_negotiation.cc:407] Connection header received
0928 22:49:43.484923 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:43.484924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:43.484938 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:43.485008 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:43.485407 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:43.485918 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:43.487099 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:43.487421 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:43.487554 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:43.496222 (+ 8668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.130:33823: BlockingRecv error: recv got EOF from 127.16.121.130:33823 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 28 22:49:43 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.131@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:43 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.131@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:43.654862 26075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:43.649845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.131:43487)
0928 22:49:43.649994 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:43.650008 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:43.650081 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:43.650265 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:43.650266 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:43.650458 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:43.650504 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:43.650507 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:43.651337 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:43.651339 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:43.651902 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:43.651904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:43.652266 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:43.652267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:43.652317 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:43.652873 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:43.652878 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:43.654788 (+ 1910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":102,"threads_started":1}
W20260928 22:49:43.655019 26073 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:43.649926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:43487 (local address 127.16.121.190:44341)
0928 22:49:43.650022 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:43.650024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:43.650083 (+ 59us) server_negotiation.cc:407] Connection header received
0928 22:49:43.650132 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:43.650132 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:43.650147 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:43.650210 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:43.650600 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:43.651202 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:43.651993 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:43.652193 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:43.652249 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:43.654938 (+ 2689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.131:43487: BlockingRecv error: recv got EOF from 127.16.121.131:43487 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 28 22:49:44 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.129@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:44 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.129@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:44.350962 26076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:44.345829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.129:39565)
0928 22:49:44.346033 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:44.346045 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:44.346104 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:44.346287 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:44.346288 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:44.346459 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:44.346506 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:44.346508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:44.347332 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:44.347334 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:44.347875 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:44.347877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:44.348256 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:44.348256 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:44.348300 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:44.348858 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:44.348864 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:44.350897 (+ 2033us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":112,"threads_started":1}
W20260928 22:49:44.351202 26077 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:44.345937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:39565 (local address 127.16.121.190:44341)
0928 22:49:44.346074 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:44.346076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:44.346088 (+ 12us) server_negotiation.cc:407] Connection header received
0928 22:49:44.346169 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:44.346169 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:44.346180 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:44.346259 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:44.346612 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:44.347221 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:44.347981 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:44.348182 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:44.348241 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:44.351115 (+ 2874us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.129:39565: BlockingRecv error: recv got EOF from 127.16.121.129:39565 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":54,"threads_started":1}
Sep 28 22:49:44 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.130@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:44 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.130@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:44.502187 26078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:44.496815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.130:57947)
0928 22:49:44.496956 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:44.496975 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:44.497036 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:44.497212 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:44.497213 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:44.497392 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:44.497473 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:44.497475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:44.498160 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:44.498162 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:44.498800 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:44.498803 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:44.499203 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:44.499204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:44.499280 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:44.499728 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:44.499733 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:44.502105 (+ 2372us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":93,"threads_started":1}
W20260928 22:49:44.502254 26077 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:44.496900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:57947 (local address 127.16.121.190:44341)
0928 22:49:44.496978 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:44.496980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:44.497075 (+ 95us) server_negotiation.cc:407] Connection header received
0928 22:49:44.497119 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:44.497119 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:44.497131 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:44.497196 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:44.497556 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:44.498062 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:44.498889 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:44.499110 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:44.499203 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:44.502193 (+ 2990us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.130:57947: BlockingRecv error: recv got EOF from 127.16.121.130:57947 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 28 22:49:44 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.131@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:44 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.131@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:44.660760 26079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:44.655417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.131:56311)
0928 22:49:44.655556 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:44.655572 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:44.655631 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:44.655860 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:44.655861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:44.656030 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:44.656077 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:44.656079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:44.656899 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:44.656901 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:44.657581 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:44.657585 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:44.658015 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:44.658016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:44.658077 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:44.658598 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:44.658609 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:44.660689 (+ 2080us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":87,"threads_started":1}
W20260928 22:49:44.660833 26077 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:44.655518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:56311 (local address 127.16.121.190:44341)
0928 22:49:44.655635 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:44.655637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:44.655685 (+ 48us) server_negotiation.cc:407] Connection header received
0928 22:49:44.655714 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:44.655715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:44.655769 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:44.655832 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:44.656192 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:44.656713 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:44.657694 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:44.657908 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:44.658014 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:44.660786 (+ 2772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.131:56311: BlockingRecv error: recv got EOF from 127.16.121.131:56311 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 28 22:49:45 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.129@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:45 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.129@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:45.356374 26080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:45.351650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.129:42801)
0928 22:49:45.351821 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:45.351832 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:45.351891 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:45.352031 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:45.352031 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:45.352211 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:45.352260 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:45.352261 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:45.352934 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:45.352934 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:45.353483 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:45.353485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:45.353834 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:45.353835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:45.353877 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:45.354363 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:45.354368 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:45.356300 (+ 1932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":98,"threads_started":1}
W20260928 22:49:45.356645 26081 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:45.351755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:42801 (local address 127.16.121.190:44341)
0928 22:49:45.351887 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:45.351889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:45.351898 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:49:45.351939 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:45.351939 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:45.351949 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:45.352008 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:45.352330 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:45.352848 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:45.353568 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:45.353765 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:45.353822 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:45.356571 (+ 2749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.129:42801: BlockingRecv error: recv got EOF from 127.16.121.129:42801 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":76,"threads_started":1}
Sep 28 22:49:45 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.130@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:45 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.130@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:45.508025 26082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:45.502847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.130:45521)
0928 22:49:45.503024 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:45.503038 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:45.503165 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:45.503333 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:45.503334 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:45.503528 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:45.503578 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:45.503580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:45.504272 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:45.504273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:45.504808 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:45.504810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:45.505244 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:45.505245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:45.505310 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:45.505856 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:45.505862 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:45.507922 (+ 2060us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":65,"threads_started":1}
W20260928 22:49:45.508080 26081 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:45.503079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:45521 (local address 127.16.121.190:44341)
0928 22:49:45.503156 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:45.503158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:45.503168 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:49:45.503234 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:45.503235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:45.503249 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:45.503327 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:45.503669 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:45.504179 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:45.504926 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:45.505130 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:45.505245 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:45.508017 (+ 2772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.130:45521: BlockingRecv error: recv got EOF from 127.16.121.130:45521 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 22:49:45 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.131@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:45 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.131@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:45.666517 26083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:45.661336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.131:42153)
0928 22:49:45.661563 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:45.661574 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:45.661633 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:45.661792 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:45.661792 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:45.661965 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:45.662008 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:45.662010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:45.662841 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:45.662843 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:45.663527 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:45.663529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:45.663910 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:45.663911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:45.663965 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:45.664606 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:45.664610 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:45.666442 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":143,"threads_started":1}
W20260928 22:49:45.666671 26081 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:45.661460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:42153 (local address 127.16.121.190:44341)
0928 22:49:45.661544 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:45.661546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:45.661635 (+ 89us) server_negotiation.cc:407] Connection header received
0928 22:49:45.661691 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:45.661692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:45.661711 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:45.661771 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:45.662117 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:45.662691 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:45.663624 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:45.663819 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:45.663877 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:45.666568 (+ 2691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.131:42153: BlockingRecv error: recv got EOF from 127.16.121.131:42153 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 22:49:46 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.129@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:46 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.129@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:46.368151 26085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:46.357025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.129:54185)
0928 22:49:46.357308 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:46.357323 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:46.357403 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:46.357600 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:46.357601 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:46.357818 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:46.357866 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:46.357876 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:46.358657 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:46.358658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:46.359223 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:46.359225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:46.359617 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:46.359618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:46.359690 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:46.365970 (+ 6280us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:46.365986 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:46.368061 (+ 2075us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":214,"threads_started":1}
W20260928 22:49:46.368317 26084 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:46.357106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:54185 (local address 127.16.121.190:44341)
0928 22:49:46.357347 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:46.357349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:46.357380 (+ 31us) server_negotiation.cc:407] Connection header received
0928 22:49:46.357500 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:46.357501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:46.357519 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:46.357595 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:46.357976 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:46.358568 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:46.359312 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:46.359529 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:46.359616 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:46.368170 (+ 8554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.129:54185: BlockingRecv error: recv got EOF from 127.16.121.129:54185 (error 108)
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":124,"threads_started":1}
Sep 28 22:49:46 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.130@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:46 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.130@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:46.513614 26086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:46.508677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.130:60909)
0928 22:49:46.508835 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:46.508845 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:46.508897 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:46.509053 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:46.509054 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:46.509240 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:46.509288 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:46.509290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:46.509987 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:46.509988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:46.510553 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:46.510555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:46.510915 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:46.510916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:46.510958 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:46.511511 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:46.511516 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:46.513554 (+ 2038us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":92,"threads_started":1}
W20260928 22:49:46.513756 26084 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:46.508784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:60909 (local address 127.16.121.190:44341)
0928 22:49:46.508838 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:46.508840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:46.508886 (+ 46us) server_negotiation.cc:407] Connection header received
0928 22:49:46.508954 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:46.508955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:46.508965 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:46.509027 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:46.509379 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:46.509904 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:46.510636 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:46.510819 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:46.510900 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:46.513674 (+ 2774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.130:60909: BlockingRecv error: recv got EOF from 127.16.121.130:60909 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 28 22:49:46 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.131@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:46 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.131@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:46.672152 26087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:46.667155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.131:42767)
0928 22:49:46.667376 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:46.667387 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:46.667479 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:46.667674 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:46.667675 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:46.667967 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:46.668029 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:46.668031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:46.668730 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:46.668731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:46.669394 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:46.669396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:46.669743 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:46.669744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:46.669795 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:46.670271 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:46.670276 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:46.672092 (+ 1816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":127,"threads_started":1}
W20260928 22:49:46.672291 26084 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:46.667258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:42767 (local address 127.16.121.190:44341)
0928 22:49:46.667321 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:46.667323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:46.667458 (+ 135us) server_negotiation.cc:407] Connection header received
0928 22:49:46.667562 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:46.667563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:46.667587 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:46.667652 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:46.668147 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:46.668655 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:46.669475 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:46.669657 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:46.669712 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:46.672212 (+ 2500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.131:42767: BlockingRecv error: recv got EOF from 127.16.121.131:42767 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 28 22:49:47 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.129@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:47 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.129@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:47.374521 26088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:47.368911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.129:41969)
0928 22:49:47.369102 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:47.369115 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:47.369188 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:47.369488 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:47.369489 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:47.369728 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:47.369791 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:47.369793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:47.370517 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:47.370518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:47.371296 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:47.371300 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:47.371727 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:47.371728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:47.371783 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:47.372419 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:47.372424 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:47.374446 (+ 2022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":95,"threads_started":1}
W20260928 22:49:47.374653 26089 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:47.369031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:41969 (local address 127.16.121.190:44341)
0928 22:49:47.369319 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:47.369321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:47.369331 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:49:47.369357 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:47.369357 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:47.369372 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:47.369449 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:47.369881 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:47.370413 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:47.371410 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:47.371618 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:47.371687 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:47.374564 (+ 2877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.129:41969: BlockingRecv error: recv got EOF from 127.16.121.129:41969 (error 108)
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":83,"threads_started":1}
Sep 28 22:49:47 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.130@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 22:49:47 dist-test-slave-56vw krb5kdc[25030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.121.130@KRBTEST.COM for oryx/127.16.121.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 22:49:47.519192 26090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:47.514172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:44341 (local address 127.16.121.130:45785)
0928 22:49:47.514304 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:47.514318 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:47.514392 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:47.514546 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:47.514546 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:47.514718 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:47.514769 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:47.514770 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:47.515451 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:47.515452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:47.516029 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:47.516030 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:47.516491 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:47.516492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:47.516540 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:47.517103 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:47.517108 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:47.519039 (+ 1931us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.121.190:44341: Server oryx/127.16.121.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
W20260928 22:49:47.519306 26089 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:47.514217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:45785 (local address 127.16.121.190:44341)
0928 22:49:47.514282 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:47.514284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:47.514435 (+ 151us) server_negotiation.cc:407] Connection header received
0928 22:49:47.514463 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:47.514464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:47.514475 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:47.514527 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:47.514838 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:47.515384 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:47.516115 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:47.516353 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:47.516479 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:47.519210 (+ 2731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.121.130:45785: BlockingRecv error: recv got EOF from 127.16.121.130:45785 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 22:49:47.567868 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 25613
I20260928 22:49:47.573017 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 25765
I20260928 22:49:47.577656 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 25899
I20260928 22:49:47.582623 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 25543
2026-09-28T22:49:47Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17861 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T22:49:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:49:47Z Disabled control of system clock
I20260928 22:49:47.616715 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:33587
--webserver_interface=127.16.121.190
--webserver_port=0
--builtin_ntp_servers=127.16.121.148:35669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:33587
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 22:49:47.699609 26094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:47.699849 26094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:47.699872 26094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:47.701215 26094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:49:47.701278 26094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:47.701292 26094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:47.701306 26094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:49:47.701318 26094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:49:47.702778 26094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:35669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:33587
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:33587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.121.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:47.703076 26094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:47.703397 26094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:47.708858 26100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:47.708856 26102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:47.708891 26099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:47.709080 26094 server_base.cc:1034] running on GCE node
I20260928 22:49:47.709331 26094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:47.709656 26094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:47.710834 26094 hybrid_clock.cc:648] HybridClock initialized: now 1790635787710822 us; error 42 us; skew 500 ppm
I20260928 22:49:47.712224 26094 webserver.cc:460] Webserver started at http://127.16.121.190:39791/ using document root <none> and password file <none>
I20260928 22:49:47.712471 26094 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:47.712543 26094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:47.712697 26094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:47.713819 26094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "7d1bf39b333b468b85e3518aa9b29b19"
format_stamp: "Formatted at 2026-09-28 22:49:47 on dist-test-slave-56vw"
server_key: "09f07924e007c51d4ec1e91ee18352ed"
server_key_iv: "262ff242389cf8f60f6f7fd086cb54d5"
server_key_version: "encryptionkey@0"
I20260928 22:49:47.714195 26094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "7d1bf39b333b468b85e3518aa9b29b19"
format_stamp: "Formatted at 2026-09-28 22:49:47 on dist-test-slave-56vw"
server_key: "09f07924e007c51d4ec1e91ee18352ed"
server_key_iv: "262ff242389cf8f60f6f7fd086cb54d5"
server_key_version: "encryptionkey@0"
I20260928 22:49:47.715641 26094 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:49:47.716614 26108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:47.716984 26094 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:47.717111 26094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "7d1bf39b333b468b85e3518aa9b29b19"
format_stamp: "Formatted at 2026-09-28 22:49:47 on dist-test-slave-56vw"
server_key: "09f07924e007c51d4ec1e91ee18352ed"
server_key_iv: "262ff242389cf8f60f6f7fd086cb54d5"
server_key_version: "encryptionkey@0"
I20260928 22:49:47.717206 26094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:47.738665 26094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:47.739199 26094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:47.739380 26094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:47.743355 26094 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.190:33587
I20260928 22:49:47.743515 26160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.190:33587 every 8 connection(s)
I20260928 22:49:47.743814 26094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:49:47.744650 26161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:47.747896 26161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d1bf39b333b468b85e3518aa9b29b19: Bootstrap starting.
I20260928 22:49:47.748564 26161 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d1bf39b333b468b85e3518aa9b29b19: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:47.748888 26161 log.cc:824] T 00000000000000000000000000000000 P 7d1bf39b333b468b85e3518aa9b29b19: Log is configured to *not* fsync() on all Append() calls
I20260928 22:49:47.749763 26161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d1bf39b333b468b85e3518aa9b29b19: No bootstrap required, opened a new log
I20260928 22:49:47.751560 26161 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d1bf39b333b468b85e3518aa9b29b19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d1bf39b333b468b85e3518aa9b29b19" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 33587 } }
I20260928 22:49:47.751713 26161 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d1bf39b333b468b85e3518aa9b29b19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:47.751735 26161 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d1bf39b333b468b85e3518aa9b29b19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d1bf39b333b468b85e3518aa9b29b19, State: Initialized, Role: FOLLOWER
I20260928 22:49:47.751899 26161 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d1bf39b333b468b85e3518aa9b29b19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d1bf39b333b468b85e3518aa9b29b19" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 33587 } }
I20260928 22:49:47.751986 26161 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d1bf39b333b468b85e3518aa9b29b19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:49:47.752009 26161 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d1bf39b333b468b85e3518aa9b29b19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:49:47.752061 26161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d1bf39b333b468b85e3518aa9b29b19 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:47.752354 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 26094
I20260928 22:49:47.752490 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:49:47.752758 16870 external_mini_cluster.cc:1428] Setting key 23da530eca2def3764ebc334cba978c7
I20260928 22:49:47.752753 26161 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d1bf39b333b468b85e3518aa9b29b19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d1bf39b333b468b85e3518aa9b29b19" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 33587 } }
I20260928 22:49:47.752871 26161 leader_election.cc:302] T 00000000000000000000000000000000 P 7d1bf39b333b468b85e3518aa9b29b19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d1bf39b333b468b85e3518aa9b29b19; no voters:
I20260928 22:49:47.753078 26161 leader_election.cc:288] T 00000000000000000000000000000000 P 7d1bf39b333b468b85e3518aa9b29b19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:49:47.753113 26164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d1bf39b333b468b85e3518aa9b29b19 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:49:47.753238 26164 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d1bf39b333b468b85e3518aa9b29b19 [term 1 LEADER]: Becoming Leader. State: Replica: 7d1bf39b333b468b85e3518aa9b29b19, State: Running, Role: LEADER
I20260928 22:49:47.753347 26164 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d1bf39b333b468b85e3518aa9b29b19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d1bf39b333b468b85e3518aa9b29b19" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 33587 } }
I20260928 22:49:47.753377 26161 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d1bf39b333b468b85e3518aa9b29b19 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:49:47.753728 26165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d1bf39b333b468b85e3518aa9b29b19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d1bf39b333b468b85e3518aa9b29b19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d1bf39b333b468b85e3518aa9b29b19" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 33587 } } }
I20260928 22:49:47.753800 26165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d1bf39b333b468b85e3518aa9b29b19 [sys.catalog]: This master's current role is: LEADER
I20260928 22:49:47.753931 26166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d1bf39b333b468b85e3518aa9b29b19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d1bf39b333b468b85e3518aa9b29b19. Latest consensus state: current_term: 1 leader_uuid: "7d1bf39b333b468b85e3518aa9b29b19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d1bf39b333b468b85e3518aa9b29b19" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 33587 } } }
I20260928 22:49:47.754035 26166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d1bf39b333b468b85e3518aa9b29b19 [sys.catalog]: This master's current role is: LEADER
I20260928 22:49:47.754496 26170 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:49:47.755065 26170 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:49:47.756811 26170 catalog_manager.cc:1347] Generated new cluster ID: 25e89c77a1884823880cb760180d2e66
I20260928 22:49:47.756865 26170 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:49:47.760232 26169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:47.753796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58604 (local address 127.16.121.190:33587)
0928 22:49:47.754259 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:47.754268 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:47.754297 (+ 29us) server_negotiation.cc:407] Connection header received
0928 22:49:47.754744 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:47.754754 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:47.754841 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:47.754986 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:47.755578 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:47.757053 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:47.757912 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:47.758432 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:47.758599 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:47.759424 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:47.759429 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:47.759429 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:49:47.759638 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:47.759672 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:49:47.759685 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:47.759784 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:47.759869 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 22:49:47.760057 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":77,"threads_started":1}
I20260928 22:49:47.762005 26170 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:49:47.762598 26170 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:49:47.769019 26170 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d1bf39b333b468b85e3518aa9b29b19: Generated new TSK 0
I20260928 22:49:47.769302 26170 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:49:47.777464 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.129:0
--local_ip_for_outbound_sockets=127.16.121.129
--webserver_interface=127.16.121.129
--webserver_port=0
--tserver_master_addrs=127.16.121.190:33587
--builtin_ntp_servers=127.16.121.148:35669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 22:49:47.864107 26185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:47.864336 26185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:47.864360 26185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:47.865664 26185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:47.865720 26185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:47.865757 26185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.129
I20260928 22:49:47.867470 26185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:35669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.121.129
--webserver_port=0
--tserver_master_addrs=127.16.121.190:33587
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.129
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:47.867776 26185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:47.868127 26185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:47.873960 26190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:47.873960 26191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:47.873977 26193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:47.874846 26185 server_base.cc:1034] running on GCE node
I20260928 22:49:47.875128 26185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:47.875432 26185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:47.876627 26185 hybrid_clock.cc:648] HybridClock initialized: now 1790635787876609 us; error 36 us; skew 500 ppm
I20260928 22:49:47.878404 26185 webserver.cc:460] Webserver started at http://127.16.121.129:36025/ using document root <none> and password file <none>
I20260928 22:49:47.878716 26185 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:47.878782 26185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:47.878927 26185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:47.880095 26185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/data/instance:
uuid: "2e3614f7cc0e496da0a106658fda1f26"
format_stamp: "Formatted at 2026-09-28 22:49:47 on dist-test-slave-56vw"
server_key: "3e91b7cbdaaa206cfe320c31a2ac00fe"
server_key_iv: "372341ca59f590ffbc2642fe7d79d2b1"
server_key_version: "encryptionkey@0"
I20260928 22:49:47.880468 26185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance:
uuid: "2e3614f7cc0e496da0a106658fda1f26"
format_stamp: "Formatted at 2026-09-28 22:49:47 on dist-test-slave-56vw"
server_key: "3e91b7cbdaaa206cfe320c31a2ac00fe"
server_key_iv: "372341ca59f590ffbc2642fe7d79d2b1"
server_key_version: "encryptionkey@0"
I20260928 22:49:47.881894 26185 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 22:49:47.882858 26199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:47.883137 26185 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:47.883276 26185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/wal
uuid: "2e3614f7cc0e496da0a106658fda1f26"
format_stamp: "Formatted at 2026-09-28 22:49:47 on dist-test-slave-56vw"
server_key: "3e91b7cbdaaa206cfe320c31a2ac00fe"
server_key_iv: "372341ca59f590ffbc2642fe7d79d2b1"
server_key_version: "encryptionkey@0"
I20260928 22:49:47.883375 26185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:47.900872 26185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:47.901261 26185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:47.901434 26185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:47.901726 26185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:49:47.902096 26185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:49:47.902155 26185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:47.902221 26185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:49:47.902251 26185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:47.909452 26185 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.129:39543
I20260928 22:49:47.909504 26312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.129:39543 every 8 connection(s)
I20260928 22:49:47.909914 26185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
I20260928 22:49:47.912917 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 26185
I20260928 22:49:47.913028 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance
I20260928 22:49:47.913226 16870 external_mini_cluster.cc:1428] Setting key 14bb9de1f0800a46d418261b88862ad4
I20260928 22:49:47.914232 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.130:0
--local_ip_for_outbound_sockets=127.16.121.130
--webserver_interface=127.16.121.130
--webserver_port=0
--tserver_master_addrs=127.16.121.190:33587
--builtin_ntp_servers=127.16.121.148:35669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 22:49:47.916069 26169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:47.910817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:42725 (local address 127.16.121.190:33587)
0928 22:49:47.910892 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:47.910895 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:47.911135 (+ 240us) server_negotiation.cc:407] Connection header received
0928 22:49:47.911767 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:47.911769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:47.911783 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:47.911844 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:47.912536 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:47.913354 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:47.914389 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:47.914610 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:47.914703 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:47.915715 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:47.915716 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:47.915717 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:49:47.915722 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:47.915739 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:49:47.915743 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:47.915817 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:47.915953 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 22:49:47.915987 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 22:49:47.916311 26315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:47.910814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:33587 (local address 127.16.121.129:42725)
0928 22:49:47.911008 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:47.911064 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:47.911635 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:47.911912 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:47.911915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:47.911924 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:47.912436 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:47.912441 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:47.913459 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:47.913460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:47.914279 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:47.914281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:47.914715 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:47.914716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:47.914882 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:47.915539 (+ 657us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:49:47.915540 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:47.915572 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:49:47.915598 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:47.915818 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:47.915824 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:47.915828 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:49:47.915935 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 22:49:47.916053 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":96,"threads_started":1}
I20260928 22:49:47.916865 26313 heartbeater.cc:347] Connected to a master server at 127.16.121.190:33587
I20260928 22:49:47.916977 26313 heartbeater.cc:464] Registering TS with master...
I20260928 22:49:47.917186 26313 heartbeater.cc:511] Master 127.16.121.190:33587 requested a full tablet report, sending...
I20260928 22:49:47.917752 26125 ts_manager.cc:194] Registered new tserver with Master: 2e3614f7cc0e496da0a106658fda1f26 (127.16.121.129:39543)
I20260928 22:49:47.918433 26125 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.121.129:42725
W20260928 22:49:48.002490 26316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:48.002731 26316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:48.002758 26316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:48.004101 26316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:48.004156 26316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:48.004185 26316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.130
I20260928 22:49:48.005677 26316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:35669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.121.130
--webserver_port=0
--tserver_master_addrs=127.16.121.190:33587
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.130
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:48.005988 26316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:48.006351 26316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:48.012444 26322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:48.012439 26321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:48.012483 26316 server_base.cc:1034] running on GCE node
W20260928 22:49:48.012511 26324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:48.012943 26316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:48.013236 26316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:48.014462 26316 hybrid_clock.cc:648] HybridClock initialized: now 1790635788014447 us; error 33 us; skew 500 ppm
I20260928 22:49:48.016069 26316 webserver.cc:460] Webserver started at http://127.16.121.130:37345/ using document root <none> and password file <none>
I20260928 22:49:48.016428 26316 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:48.016511 26316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:48.016670 26316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:48.017813 26316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/data/instance:
uuid: "32f0f45a4d6a41d797b95e37454abf61"
format_stamp: "Formatted at 2026-09-28 22:49:48 on dist-test-slave-56vw"
server_key: "7258d72ae35c18b15b3e48a08a5d425a"
server_key_iv: "eb724055363c99f17a8b5cd9d5f45d87"
server_key_version: "encryptionkey@0"
I20260928 22:49:48.018198 26316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance:
uuid: "32f0f45a4d6a41d797b95e37454abf61"
format_stamp: "Formatted at 2026-09-28 22:49:48 on dist-test-slave-56vw"
server_key: "7258d72ae35c18b15b3e48a08a5d425a"
server_key_iv: "eb724055363c99f17a8b5cd9d5f45d87"
server_key_version: "encryptionkey@0"
I20260928 22:49:48.019817 26316 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:49:48.020715 26330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:48.020967 26316 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 22:49:48.021087 26316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/wal
uuid: "32f0f45a4d6a41d797b95e37454abf61"
format_stamp: "Formatted at 2026-09-28 22:49:48 on dist-test-slave-56vw"
server_key: "7258d72ae35c18b15b3e48a08a5d425a"
server_key_iv: "eb724055363c99f17a8b5cd9d5f45d87"
server_key_version: "encryptionkey@0"
I20260928 22:49:48.021169 26316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:48.028445 26316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:48.028764 26316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:48.028918 26316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:48.029135 26316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:49:48.029479 26316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:49:48.029537 26316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:48.029578 26316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:49:48.029605 26316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:48.035902 26316 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.130:33937
I20260928 22:49:48.035938 26443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.130:33937 every 8 connection(s)
I20260928 22:49:48.036366 26316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
I20260928 22:49:48.040119 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 26316
I20260928 22:49:48.040253 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance
I20260928 22:49:48.040454 16870 external_mini_cluster.cc:1428] Setting key 5872fd00c976329b7114628aa0776870
I20260928 22:49:48.041566 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.131:0
--local_ip_for_outbound_sockets=127.16.121.131
--webserver_interface=127.16.121.131
--webserver_port=0
--tserver_master_addrs=127.16.121.190:33587
--builtin_ntp_servers=127.16.121.148:35669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 22:49:48.043432 26169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:48.037252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:48551 (local address 127.16.121.190:33587)
0928 22:49:48.037406 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:48.037412 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:48.037609 (+ 197us) server_negotiation.cc:407] Connection header received
0928 22:49:48.038152 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:48.038153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:48.038165 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:48.038344 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:48.039247 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:48.039795 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:48.041427 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:48.041765 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:48.041840 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:48.043032 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:48.043033 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:48.043034 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:49:48.043064 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:48.043083 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:49:48.043089 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:48.043166 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:48.043281 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 22:49:48.043317 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260928 22:49:48.043572 26446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:48.037235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:33587 (local address 127.16.121.130:48551)
0928 22:49:48.037465 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:48.037511 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:48.038026 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:48.038487 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:48.038491 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:48.038501 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:48.039069 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:48.039074 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:48.039948 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:48.039949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:48.041277 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:48.041280 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:48.041920 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:48.041921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:48.042112 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:48.042773 (+ 661us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:49:48.042773 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:48.042818 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:49:48.042852 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:48.043165 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:48.043174 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:48.043178 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:49:48.043255 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 22:49:48.043355 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":114,"threads_started":1}
I20260928 22:49:48.044148 26444 heartbeater.cc:347] Connected to a master server at 127.16.121.190:33587
I20260928 22:49:48.044261 26444 heartbeater.cc:464] Registering TS with master...
I20260928 22:49:48.044483 26444 heartbeater.cc:511] Master 127.16.121.190:33587 requested a full tablet report, sending...
I20260928 22:49:48.044962 26125 ts_manager.cc:194] Registered new tserver with Master: 32f0f45a4d6a41d797b95e37454abf61 (127.16.121.130:33937)
I20260928 22:49:48.045356 26125 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.121.130:48551
W20260928 22:49:48.127954 26447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:48.128189 26447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:48.128213 26447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:48.129822 26447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:48.129931 26447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:48.129963 26447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.131
I20260928 22:49:48.131696 26447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:35669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.121.131
--webserver_port=0
--tserver_master_addrs=127.16.121.190:33587
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.131
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:48.132015 26447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:48.132351 26447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:48.138965 26455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:48.139173 26453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:48.139233 26452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:48.139364 26447 server_base.cc:1034] running on GCE node
I20260928 22:49:48.139719 26447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:48.140030 26447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:48.141229 26447 hybrid_clock.cc:648] HybridClock initialized: now 1790635788141202 us; error 38 us; skew 500 ppm
I20260928 22:49:48.143189 26447 webserver.cc:460] Webserver started at http://127.16.121.131:42905/ using document root <none> and password file <none>
I20260928 22:49:48.143709 26447 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:48.143800 26447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:48.143970 26447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:48.145772 26447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/data/instance:
uuid: "5af01d6153634be9a45bbda29b88552a"
format_stamp: "Formatted at 2026-09-28 22:49:48 on dist-test-slave-56vw"
server_key: "ca37bc2cbc31a37286f8bfaae6ccc911"
server_key_iv: "25315954d0fafefe77f76e5bbd00cabb"
server_key_version: "encryptionkey@0"
I20260928 22:49:48.146204 26447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance:
uuid: "5af01d6153634be9a45bbda29b88552a"
format_stamp: "Formatted at 2026-09-28 22:49:48 on dist-test-slave-56vw"
server_key: "ca37bc2cbc31a37286f8bfaae6ccc911"
server_key_iv: "25315954d0fafefe77f76e5bbd00cabb"
server_key_version: "encryptionkey@0"
I20260928 22:49:48.147859 26447 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.003s
I20260928 22:49:48.148898 26461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:48.149176 26447 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:48.149294 26447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/wal
uuid: "5af01d6153634be9a45bbda29b88552a"
format_stamp: "Formatted at 2026-09-28 22:49:48 on dist-test-slave-56vw"
server_key: "ca37bc2cbc31a37286f8bfaae6ccc911"
server_key_iv: "25315954d0fafefe77f76e5bbd00cabb"
server_key_version: "encryptionkey@0"
I20260928 22:49:48.149475 26447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:48.170800 26447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:48.171180 26447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:48.171357 26447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:48.171607 26447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:49:48.171958 26447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:49:48.172016 26447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:48.172058 26447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:49:48.172103 26447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:48.178174 26447 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.131:41881
I20260928 22:49:48.178229 26574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.131:41881 every 8 connection(s)
I20260928 22:49:48.178612 26447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
I20260928 22:49:48.185106 26169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:48.179616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:36547 (local address 127.16.121.190:33587)
0928 22:49:48.179664 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:48.179666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:48.180051 (+ 385us) server_negotiation.cc:407] Connection header received
0928 22:49:48.180470 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:48.180472 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:48.180485 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:48.180560 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:48.181618 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:48.182209 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:48.183555 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:48.183748 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:48.183837 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:48.184838 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:48.184840 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:48.184840 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:49:48.184846 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:48.184861 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:49:48.184866 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:48.184922 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:48.185008 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 22:49:48.185039 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 22:49:48.185355 26577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:48.179534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:33587 (local address 127.16.121.131:36547)
0928 22:49:48.179735 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:48.179781 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:48.180310 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:48.180710 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:48.180715 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:48.180724 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:48.181331 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:48.181347 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:48.182338 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:48.182339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:48.183443 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:48.183446 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:48.183877 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:48.183878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:48.184029 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:48.184670 (+ 641us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:49:48.184671 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:48.184704 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:49:48.184728 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:48.184931 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:48.184935 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:48.184938 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:49:48.184996 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 22:49:48.185089 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":92,"threads_started":1}
I20260928 22:49:48.186079 26575 heartbeater.cc:347] Connected to a master server at 127.16.121.190:33587
I20260928 22:49:48.186215 26575 heartbeater.cc:464] Registering TS with master...
I20260928 22:49:48.186437 26575 heartbeater.cc:511] Master 127.16.121.190:33587 requested a full tablet report, sending...
I20260928 22:49:48.186966 26125 ts_manager.cc:194] Registered new tserver with Master: 5af01d6153634be9a45bbda29b88552a (127.16.121.131:41881)
I20260928 22:49:48.187441 26125 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.121.131:36547
I20260928 22:49:48.188349 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 26447
I20260928 22:49:48.188472 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance
I20260928 22:49:48.188809 16870 external_mini_cluster.cc:1428] Setting key e01d9606961b8958acd29580cce6e33b
I20260928 22:49:48.190232 16870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 22:49:48.192695 26169 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:48.191813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58610 (local address 127.16.121.190:33587)
0928 22:49:48.192064 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:48.192066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:48.192076 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:49:48.192110 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:48.192110 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:48.192130 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:48.192190 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:48.192639 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58610: BlockingRecv error: recv got EOF from 127.0.0.1:58610 (error 108)
Metrics: {"server-negotiator.queue_time_us":220}
I20260928 22:49:48.192901 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 26185
I20260928 22:49:48.198206 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 26316
I20260928 22:49:48.203744 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 26447
I20260928 22:49:48.208777 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 26094
2026-09-28T22:49:48Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (623 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T22:49:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:49:48Z Disabled control of system clock
I20260928 22:49:48.237888 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:33117
--webserver_interface=127.16.121.190
--webserver_port=0
--builtin_ntp_servers=127.16.121.148:33967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:33117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 22:49:48.335806 26586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:48.336058 26586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:48.336083 26586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:48.337484 26586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:49:48.337555 26586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:48.337569 26586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:48.337584 26586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:49:48.337595 26586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:49:48.339581 26586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:33967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:33117
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:33117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.121.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:48.340060 26586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:48.340466 26586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:48.348165 26594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:48.348297 26592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:48.348165 26591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:48.349606 26586 server_base.cc:1034] running on GCE node
I20260928 22:49:48.349944 26586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:48.350279 26586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:48.351634 26586 hybrid_clock.cc:648] HybridClock initialized: now 1790635788351583 us; error 63 us; skew 500 ppm
I20260928 22:49:48.353473 26586 webserver.cc:460] Webserver started at http://127.16.121.190:41613/ using document root <none> and password file <none>
I20260928 22:49:48.353771 26586 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:48.354099 26586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:48.354477 26586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:48.355844 26586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "cb4ea1cb43e04a3995d028283a80c647"
format_stamp: "Formatted at 2026-09-28 22:49:48 on dist-test-slave-56vw"
server_key: "1ede172c2c6a42666540bf8333726053"
server_key_iv: "663146f362a71081ab0846f56d14181c"
server_key_version: "encryptionkey@0"
I20260928 22:49:48.356248 26586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "cb4ea1cb43e04a3995d028283a80c647"
format_stamp: "Formatted at 2026-09-28 22:49:48 on dist-test-slave-56vw"
server_key: "1ede172c2c6a42666540bf8333726053"
server_key_iv: "663146f362a71081ab0846f56d14181c"
server_key_version: "encryptionkey@0"
I20260928 22:49:48.358644 26586 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 22:49:48.360248 26600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:48.360592 26586 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:48.360705 26586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "cb4ea1cb43e04a3995d028283a80c647"
format_stamp: "Formatted at 2026-09-28 22:49:48 on dist-test-slave-56vw"
server_key: "1ede172c2c6a42666540bf8333726053"
server_key_iv: "663146f362a71081ab0846f56d14181c"
server_key_version: "encryptionkey@0"
I20260928 22:49:48.360767 26586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:48.365505 26586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:48.365983 26586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:48.366207 26586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:48.371953 26586 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.190:33117
I20260928 22:49:48.372103 26652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.190:33117 every 8 connection(s)
I20260928 22:49:48.372467 26586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:49:48.373410 26653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:48.374065 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 26586
I20260928 22:49:48.374173 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:49:48.374433 16870 external_mini_cluster.cc:1428] Setting key 34f43d060640684c4f6a95a919584a79
I20260928 22:49:48.378523 26656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:48.375217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48718 (local address 127.16.121.190:33117)
0928 22:49:48.375458 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:48.375466 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:48.375497 (+ 31us) server_negotiation.cc:407] Connection header received
0928 22:49:48.376255 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:48.376265 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:48.376413 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:48.376775 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:48.378047 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:48.378058 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:48.378058 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:49:48.378063 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:48.378116 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:49:48.378128 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:48.378205 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:48.378299 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 22:49:48.378365 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":95,"threads_started":1}
I20260928 22:49:48.378755 26653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb4ea1cb43e04a3995d028283a80c647: Bootstrap starting.
I20260928 22:49:48.379719 26653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb4ea1cb43e04a3995d028283a80c647: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:48.380069 26653 log.cc:824] T 00000000000000000000000000000000 P cb4ea1cb43e04a3995d028283a80c647: Log is configured to *not* fsync() on all Append() calls
I20260928 22:49:48.381582 26653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb4ea1cb43e04a3995d028283a80c647: No bootstrap required, opened a new log
I20260928 22:49:48.384292 26653 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb4ea1cb43e04a3995d028283a80c647 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb4ea1cb43e04a3995d028283a80c647" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 33117 } }
I20260928 22:49:48.384495 26653 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb4ea1cb43e04a3995d028283a80c647 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:48.384546 26653 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb4ea1cb43e04a3995d028283a80c647 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb4ea1cb43e04a3995d028283a80c647, State: Initialized, Role: FOLLOWER
I20260928 22:49:48.384681 26653 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb4ea1cb43e04a3995d028283a80c647 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb4ea1cb43e04a3995d028283a80c647" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 33117 } }
I20260928 22:49:48.384768 26653 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb4ea1cb43e04a3995d028283a80c647 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:49:48.384809 26653 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb4ea1cb43e04a3995d028283a80c647 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:49:48.384860 26653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb4ea1cb43e04a3995d028283a80c647 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:48.385524 26653 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb4ea1cb43e04a3995d028283a80c647 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb4ea1cb43e04a3995d028283a80c647" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 33117 } }
I20260928 22:49:48.385879 26653 leader_election.cc:302] T 00000000000000000000000000000000 P cb4ea1cb43e04a3995d028283a80c647 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb4ea1cb43e04a3995d028283a80c647; no voters:
I20260928 22:49:48.386157 26653 leader_election.cc:288] T 00000000000000000000000000000000 P cb4ea1cb43e04a3995d028283a80c647 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:49:48.386478 26653 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb4ea1cb43e04a3995d028283a80c647 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:49:48.386715 26658 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb4ea1cb43e04a3995d028283a80c647 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:49:48.386909 26658 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb4ea1cb43e04a3995d028283a80c647 [term 1 LEADER]: Becoming Leader. State: Replica: cb4ea1cb43e04a3995d028283a80c647, State: Running, Role: LEADER
I20260928 22:49:48.387051 26658 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb4ea1cb43e04a3995d028283a80c647 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb4ea1cb43e04a3995d028283a80c647" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 33117 } }
I20260928 22:49:48.388181 26660 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb4ea1cb43e04a3995d028283a80c647 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb4ea1cb43e04a3995d028283a80c647. Latest consensus state: current_term: 1 leader_uuid: "cb4ea1cb43e04a3995d028283a80c647" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb4ea1cb43e04a3995d028283a80c647" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 33117 } } }
I20260928 22:49:48.388276 26660 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb4ea1cb43e04a3995d028283a80c647 [sys.catalog]: This master's current role is: LEADER
I20260928 22:49:48.388376 26659 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb4ea1cb43e04a3995d028283a80c647 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb4ea1cb43e04a3995d028283a80c647" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb4ea1cb43e04a3995d028283a80c647" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 33117 } } }
I20260928 22:49:48.388811 26659 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb4ea1cb43e04a3995d028283a80c647 [sys.catalog]: This master's current role is: LEADER
W20260928 22:49:48.388976 26672 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cb4ea1cb43e04a3995d028283a80c647: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:49:48.389036 26672 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:49:48.389094 26674 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:49:48.389274 26674 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:49:48.395270 26674 catalog_manager.cc:1347] Generated new cluster ID: d5b5f3044dd14d579a673721e3205034
I20260928 22:49:48.395360 26674 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:49:48.429626 26674 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:49:48.429886 26674 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:49:48.438546 26674 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb4ea1cb43e04a3995d028283a80c647: Generated new TSK 0
I20260928 22:49:48.438829 26674 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:49:48.441224 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.129:0
--local_ip_for_outbound_sockets=127.16.121.129
--webserver_interface=127.16.121.129
--webserver_port=0
--tserver_master_addrs=127.16.121.190:33117
--builtin_ntp_servers=127.16.121.148:33967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 22:49:48.530954 26678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:48.531258 26678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:48.531301 26678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:48.532634 26678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:48.532711 26678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:48.532758 26678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.129
I20260928 22:49:48.534299 26678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:33967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.121.129
--webserver_port=0
--tserver_master_addrs=127.16.121.190:33117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.129
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:48.534931 26678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:48.535475 26678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:48.541674 26684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:48.541685 26686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:48.541697 26683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:48.541982 26678 server_base.cc:1034] running on GCE node
I20260928 22:49:48.542616 26678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:48.542941 26678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:48.544130 26678 hybrid_clock.cc:648] HybridClock initialized: now 1790635788544071 us; error 71 us; skew 500 ppm
I20260928 22:49:48.545825 26678 webserver.cc:460] Webserver started at http://127.16.121.129:33243/ using document root <none> and password file <none>
I20260928 22:49:48.546128 26678 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:48.546188 26678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:48.546353 26678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:48.547621 26678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/data/instance:
uuid: "3049e74bc579410a82af74e22a49dee2"
format_stamp: "Formatted at 2026-09-28 22:49:48 on dist-test-slave-56vw"
server_key: "2619586809e8fb1aa957ddd000cd58a6"
server_key_iv: "cebf6c20e1682d55707cfe4c5927a214"
server_key_version: "encryptionkey@0"
I20260928 22:49:48.548020 26678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance:
uuid: "3049e74bc579410a82af74e22a49dee2"
format_stamp: "Formatted at 2026-09-28 22:49:48 on dist-test-slave-56vw"
server_key: "2619586809e8fb1aa957ddd000cd58a6"
server_key_iv: "cebf6c20e1682d55707cfe4c5927a214"
server_key_version: "encryptionkey@0"
I20260928 22:49:48.549506 26678 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:49:48.550379 26692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:48.550676 26678 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:48.550765 26678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/wal
uuid: "3049e74bc579410a82af74e22a49dee2"
format_stamp: "Formatted at 2026-09-28 22:49:48 on dist-test-slave-56vw"
server_key: "2619586809e8fb1aa957ddd000cd58a6"
server_key_iv: "cebf6c20e1682d55707cfe4c5927a214"
server_key_version: "encryptionkey@0"
I20260928 22:49:48.550827 26678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:48.554312 26678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:48.554556 26678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:48.554652 26678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:48.554818 26678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:49:48.555125 26678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:49:48.555161 26678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:48.555182 26678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:49:48.555195 26678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:48.561069 26678 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.129:43539
I20260928 22:49:48.561161 26805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.129:43539 every 8 connection(s)
I20260928 22:49:48.561496 26678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
I20260928 22:49:48.564738 26656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:48.562301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:56579 (local address 127.16.121.190:33117)
0928 22:49:48.562397 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:48.562400 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:48.562603 (+ 203us) server_negotiation.cc:407] Connection header received
0928 22:49:48.563188 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:48.563189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:48.563195 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:48.563252 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:48.564488 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:48.564489 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:48.564490 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:49:48.564494 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:48.564507 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:49:48.564512 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:48.564560 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:48.564670 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 22:49:48.564682 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 22:49:48.564879 26808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:48.562256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:33117 (local address 127.16.121.129:56579)
0928 22:49:48.562480 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:48.562515 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:48.563031 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:48.563314 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:48.563318 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:48.563327 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:48.564322 (+ 995us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:49:48.564323 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:48.564354 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:49:48.564403 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:48.564560 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:48.564562 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:48.564565 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:49:48.564618 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 22:49:48.564670 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":112,"threads_started":1}
I20260928 22:49:48.565395 26806 heartbeater.cc:347] Connected to a master server at 127.16.121.190:33117
I20260928 22:49:48.565483 26806 heartbeater.cc:464] Registering TS with master...
I20260928 22:49:48.565595 26806 heartbeater.cc:511] Master 127.16.121.190:33117 requested a full tablet report, sending...
I20260928 22:49:48.566082 26617 ts_manager.cc:194] Registered new tserver with Master: 3049e74bc579410a82af74e22a49dee2 (127.16.121.129:43539)
I20260928 22:49:48.566692 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 26678
I20260928 22:49:48.566785 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance
I20260928 22:49:48.566933 16870 external_mini_cluster.cc:1428] Setting key 0c33724223c2d130837df7fa2ae7728c
I20260928 22:49:48.568151 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.130:0
--local_ip_for_outbound_sockets=127.16.121.130
--webserver_interface=127.16.121.130
--webserver_port=0
--tserver_master_addrs=127.16.121.190:33117
--builtin_ntp_servers=127.16.121.148:33967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 22:49:48.663097 26809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:48.663384 26809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:48.663412 26809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:48.664866 26809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:48.664917 26809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:48.664950 26809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.130
I20260928 22:49:48.666579 26809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:33967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.121.130
--webserver_port=0
--tserver_master_addrs=127.16.121.190:33117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.130
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:48.666821 26809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:48.667136 26809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:48.672994 26817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:48.673002 26814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:48.673035 26815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:48.673252 26809 server_base.cc:1034] running on GCE node
I20260928 22:49:48.673640 26809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:48.673883 26809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:48.675151 26809 hybrid_clock.cc:648] HybridClock initialized: now 1790635788675129 us; error 35 us; skew 500 ppm
I20260928 22:49:48.676743 26809 webserver.cc:460] Webserver started at http://127.16.121.130:45343/ using document root <none> and password file <none>
I20260928 22:49:48.677045 26809 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:48.677126 26809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:48.677294 26809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:48.678529 26809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/data/instance:
uuid: "68c19a6f89a34c9692f40f41ccce5b59"
format_stamp: "Formatted at 2026-09-28 22:49:48 on dist-test-slave-56vw"
server_key: "6d4f31493df6245746da892dd450a667"
server_key_iv: "61c55f27edf4ae4c0bb724a4df8df80a"
server_key_version: "encryptionkey@0"
I20260928 22:49:48.678937 26809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance:
uuid: "68c19a6f89a34c9692f40f41ccce5b59"
format_stamp: "Formatted at 2026-09-28 22:49:48 on dist-test-slave-56vw"
server_key: "6d4f31493df6245746da892dd450a667"
server_key_iv: "61c55f27edf4ae4c0bb724a4df8df80a"
server_key_version: "encryptionkey@0"
I20260928 22:49:48.680546 26809 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:49:48.681381 26823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:48.681602 26809 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:49:48.681726 26809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/wal
uuid: "68c19a6f89a34c9692f40f41ccce5b59"
format_stamp: "Formatted at 2026-09-28 22:49:48 on dist-test-slave-56vw"
server_key: "6d4f31493df6245746da892dd450a667"
server_key_iv: "61c55f27edf4ae4c0bb724a4df8df80a"
server_key_version: "encryptionkey@0"
I20260928 22:49:48.681819 26809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:48.685989 26809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:48.686300 26809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:48.686467 26809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:48.686702 26809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:49:48.687127 26809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:49:48.687194 26809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:48.687242 26809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:49:48.687271 26809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:48.694002 26809 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.130:34973
I20260928 22:49:48.694046 26936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.130:34973 every 8 connection(s)
I20260928 22:49:48.694412 26809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
I20260928 22:49:48.697696 26656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:48.695236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:50077 (local address 127.16.121.190:33117)
0928 22:49:48.695312 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:48.695315 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:48.695600 (+ 285us) server_negotiation.cc:407] Connection header received
0928 22:49:48.696091 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:48.696093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:48.696099 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:48.696160 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:48.697464 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:48.697466 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:48.697467 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:49:48.697471 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:48.697488 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:49:48.697493 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:48.697562 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:48.697621 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 22:49:48.697635 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 22:49:48.697921 26939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:48.695272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:33117 (local address 127.16.121.130:50077)
0928 22:49:48.695469 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:48.695503 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:48.695972 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:48.696240 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:48.696244 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:48.696252 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:48.697293 (+ 1041us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:49:48.697294 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:48.697325 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:49:48.697356 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:48.697562 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:48.697565 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:48.697568 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:49:48.697617 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 22:49:48.697666 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":84,"threads_started":1}
I20260928 22:49:48.698520 26937 heartbeater.cc:347] Connected to a master server at 127.16.121.190:33117
I20260928 22:49:48.698597 26937 heartbeater.cc:464] Registering TS with master...
I20260928 22:49:48.698763 26937 heartbeater.cc:511] Master 127.16.121.190:33117 requested a full tablet report, sending...
I20260928 22:49:48.699162 26617 ts_manager.cc:194] Registered new tserver with Master: 68c19a6f89a34c9692f40f41ccce5b59 (127.16.121.130:34973)
I20260928 22:49:48.703805 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 26809
I20260928 22:49:48.703974 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance
I20260928 22:49:48.704157 16870 external_mini_cluster.cc:1428] Setting key 47651b6317dc0e7d6cf0a307fe7a8c4d
I20260928 22:49:48.705202 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.131:0
--local_ip_for_outbound_sockets=127.16.121.131
--webserver_interface=127.16.121.131
--webserver_port=0
--tserver_master_addrs=127.16.121.190:33117
--builtin_ntp_servers=127.16.121.148:33967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 22:49:48.788324 26940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:48.788545 26940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:48.788568 26940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:48.789774 26940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:48.789816 26940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:48.789844 26940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.131
I20260928 22:49:48.791303 26940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:33967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.121.131
--webserver_port=0
--tserver_master_addrs=127.16.121.190:33117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.131
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:48.791575 26940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:48.791877 26940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:48.797133 26946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:48.797065 26948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:48.797062 26945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:48.797071 26940 server_base.cc:1034] running on GCE node
I20260928 22:49:48.797489 26940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:48.797689 26940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:48.798842 26940 hybrid_clock.cc:648] HybridClock initialized: now 1790635788798825 us; error 40 us; skew 500 ppm
I20260928 22:49:48.800312 26940 webserver.cc:460] Webserver started at http://127.16.121.131:35309/ using document root <none> and password file <none>
I20260928 22:49:48.800630 26940 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:48.800707 26940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:48.800858 26940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:48.801963 26940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/data/instance:
uuid: "9265efa7204249419c722203b439d981"
format_stamp: "Formatted at 2026-09-28 22:49:48 on dist-test-slave-56vw"
server_key: "6433f138143f356f67d0c441332bffaf"
server_key_iv: "7f7d5fad2a68991f4843c42789cceebe"
server_key_version: "encryptionkey@0"
I20260928 22:49:48.802336 26940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance:
uuid: "9265efa7204249419c722203b439d981"
format_stamp: "Formatted at 2026-09-28 22:49:48 on dist-test-slave-56vw"
server_key: "6433f138143f356f67d0c441332bffaf"
server_key_iv: "7f7d5fad2a68991f4843c42789cceebe"
server_key_version: "encryptionkey@0"
I20260928 22:49:48.803759 26940 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 22:49:48.804556 26954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:48.804759 26940 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:48.804867 26940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/wal
uuid: "9265efa7204249419c722203b439d981"
format_stamp: "Formatted at 2026-09-28 22:49:48 on dist-test-slave-56vw"
server_key: "6433f138143f356f67d0c441332bffaf"
server_key_iv: "7f7d5fad2a68991f4843c42789cceebe"
server_key_version: "encryptionkey@0"
I20260928 22:49:48.804943 26940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:48.809098 26940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:48.809507 26940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:48.809684 26940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:48.809924 26940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:49:48.810281 26940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:49:48.810348 26940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:48.810392 26940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:49:48.810423 26940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:48.817487 26940 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.131:39939
I20260928 22:49:48.817536 27067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.131:39939 every 8 connection(s)
I20260928 22:49:48.817925 26940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
I20260928 22:49:48.820309 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 26940
I20260928 22:49:48.820461 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance
I20260928 22:49:48.820662 16870 external_mini_cluster.cc:1428] Setting key 4e19db123e151f454dfaee6b1901d585
I20260928 22:49:48.821544 26656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:48.818714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:57497 (local address 127.16.121.190:33117)
0928 22:49:48.818806 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:48.818809 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:48.819029 (+ 220us) server_negotiation.cc:407] Connection header received
0928 22:49:48.819615 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:48.819617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:48.819623 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:48.819684 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:48.821128 (+ 1444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:48.821130 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:48.821130 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:49:48.821134 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:48.821167 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:49:48.821173 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:48.821238 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:48.821445 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 22:49:48.821462 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 22:49:48.821674 27070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:48.818664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:33117 (local address 127.16.121.131:57497)
0928 22:49:48.818889 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:48.818927 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:48.819510 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:48.819747 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:48.819751 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:48.819760 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:48.820953 (+ 1193us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:49:48.820954 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:48.820986 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:49:48.821017 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:48.821294 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:48.821299 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:48.821302 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:49:48.821369 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 22:49:48.821431 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":108,"threads_started":1}
I20260928 22:49:48.822171 27068 heartbeater.cc:347] Connected to a master server at 127.16.121.190:33117
I20260928 22:49:48.822304 27068 heartbeater.cc:464] Registering TS with master...
I20260928 22:49:48.822530 27068 heartbeater.cc:511] Master 127.16.121.190:33117 requested a full tablet report, sending...
I20260928 22:49:48.822945 26617 ts_manager.cc:194] Registered new tserver with Master: 9265efa7204249419c722203b439d981 (127.16.121.131:39939)
I20260928 22:49:48.833300 16870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 22:49:48.835587 26656 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:49:48.834911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48720 (local address 127.16.121.190:33117)
0928 22:49:48.834985 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:48.834987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:48.835135 (+ 148us) server_negotiation.cc:407] Connection header received
0928 22:49:48.835270 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:48.835271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:48.835278 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:48.835348 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:48.835496 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48720: BlockingRecv error: recv got EOF from 127.0.0.1:48720 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 22:49:48.835706 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 26678
I20260928 22:49:48.840592 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 26809
I20260928 22:49:48.844846 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 26940
I20260928 22:49:48.849246 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 26586
2026-09-28T22:49:48Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (639 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:49:48 dist-test-slave-56vw krb5kdc[27078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:49:48 dist-test-slave-56vw krb5kdc[27078](info): set up 2 sockets
Sep 28 22:49:48 dist-test-slave-56vw krb5kdc[27078](info): commencing operation
krb5kdc: starting...
W20260928 22:49:49.924125 16870 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Sep 28 22:49:50 dist-test-slave-56vw krb5kdc[27078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:49:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:49:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.121.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
I20260928 22:49:50.055894 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:43911
--webserver_interface=127.16.121.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:36469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:43911
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:49:50.138484 27091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:50.138705 27091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:50.138728 27091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:50.140038 27091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:49:50.140089 27091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:50.140105 27091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:50.140116 27091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:49:50.140128 27091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:49:50.141775 27091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:36469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:43911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:43911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.121.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:50.142045 27091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:50.142323 27091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:50.147523 27096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:50.147753 27097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:50.147686 27099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:50.147790 27091 server_base.cc:1034] running on GCE node
I20260928 22:49:50.148167 27091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:50.148512 27091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:50.149667 27091 hybrid_clock.cc:648] HybridClock initialized: now 1790635790149647 us; error 43 us; skew 500 ppm
Sep 28 22:49:50 dist-test-slave-56vw krb5kdc[27078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635790, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:50.152673 27091 init.cc:377] Logged in from keytab as kudu/127.16.121.190@KRBTEST.COM (short username kudu)
I20260928 22:49:50.153184 27091 webserver.cc:460] Webserver started at http://127.16.121.190:41891/ using document root <none> and password file <none>
I20260928 22:49:50.153440 27091 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:50.153488 27091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:50.153636 27091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:50.154789 27091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "aa38796cf78e4a8b9ef9c859e02922bf"
format_stamp: "Formatted at 2026-09-28 22:49:50 on dist-test-slave-56vw"
server_key: "c51f6604eedf79bfd75c7cedd7eb770b"
server_key_iv: "0becacef34f1f4db6e8af5609abff6df"
server_key_version: "encryptionkey@0"
I20260928 22:49:50.155195 27091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "aa38796cf78e4a8b9ef9c859e02922bf"
format_stamp: "Formatted at 2026-09-28 22:49:50 on dist-test-slave-56vw"
server_key: "c51f6604eedf79bfd75c7cedd7eb770b"
server_key_iv: "0becacef34f1f4db6e8af5609abff6df"
server_key_version: "encryptionkey@0"
I20260928 22:49:50.156613 27091 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:49:50.157472 27106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:50.157720 27091 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:49:50.157830 27091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "aa38796cf78e4a8b9ef9c859e02922bf"
format_stamp: "Formatted at 2026-09-28 22:49:50 on dist-test-slave-56vw"
server_key: "c51f6604eedf79bfd75c7cedd7eb770b"
server_key_iv: "0becacef34f1f4db6e8af5609abff6df"
server_key_version: "encryptionkey@0"
I20260928 22:49:50.157914 27091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:50.173357 27091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:50.175731 27091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:50.175886 27091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:50.180478 27091 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.190:43911
I20260928 22:49:50.180558 27158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.190:43911 every 8 connection(s)
I20260928 22:49:50.180943 27091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:49:50.181161 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 27091
I20260928 22:49:50.181289 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:49:50.181510 16870 external_mini_cluster.cc:1428] Setting key ef354c2ec4f55395fd7656c7fdc15d21
I20260928 22:49:50.182495 27159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:50.185973 27159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa38796cf78e4a8b9ef9c859e02922bf: Bootstrap starting.
I20260928 22:49:50.186712 27159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa38796cf78e4a8b9ef9c859e02922bf: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:50.187094 27159 log.cc:824] T 00000000000000000000000000000000 P aa38796cf78e4a8b9ef9c859e02922bf: Log is configured to *not* fsync() on all Append() calls
I20260928 22:49:50.188028 27159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa38796cf78e4a8b9ef9c859e02922bf: No bootstrap required, opened a new log
Sep 28 22:49:50 dist-test-slave-56vw krb5kdc[27078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:49:50.190668 27159 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa38796cf78e4a8b9ef9c859e02922bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa38796cf78e4a8b9ef9c859e02922bf" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 43911 } }
I20260928 22:49:50.190882 27159 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa38796cf78e4a8b9ef9c859e02922bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:50.190927 27159 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa38796cf78e4a8b9ef9c859e02922bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa38796cf78e4a8b9ef9c859e02922bf, State: Initialized, Role: FOLLOWER
I20260928 22:49:50.191124 27159 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa38796cf78e4a8b9ef9c859e02922bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa38796cf78e4a8b9ef9c859e02922bf" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 43911 } }
I20260928 22:49:50.191247 27159 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa38796cf78e4a8b9ef9c859e02922bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:49:50.191279 27159 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa38796cf78e4a8b9ef9c859e02922bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:49:50.191313 27159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa38796cf78e4a8b9ef9c859e02922bf [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:50.192134 27159 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa38796cf78e4a8b9ef9c859e02922bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa38796cf78e4a8b9ef9c859e02922bf" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 43911 } }
I20260928 22:49:50.192235 27159 leader_election.cc:302] T 00000000000000000000000000000000 P aa38796cf78e4a8b9ef9c859e02922bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa38796cf78e4a8b9ef9c859e02922bf; no voters:
I20260928 22:49:50.192464 27159 leader_election.cc:288] T 00000000000000000000000000000000 P aa38796cf78e4a8b9ef9c859e02922bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:49:50.192523 27164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa38796cf78e4a8b9ef9c859e02922bf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:49:50.192737 27164 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa38796cf78e4a8b9ef9c859e02922bf [term 1 LEADER]: Becoming Leader. State: Replica: aa38796cf78e4a8b9ef9c859e02922bf, State: Running, Role: LEADER
I20260928 22:49:50.192840 27159 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa38796cf78e4a8b9ef9c859e02922bf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:49:50.194396 27164 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa38796cf78e4a8b9ef9c859e02922bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa38796cf78e4a8b9ef9c859e02922bf" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 43911 } }
I20260928 22:49:50.195274 27162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:50.182139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56590 (local address 127.16.121.190:43911)
0928 22:49:50.182377 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:50.182387 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:50.182424 (+ 37us) server_negotiation.cc:407] Connection header received
0928 22:49:50.183170 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:50.183180 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:50.183262 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:50.183436 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:50.184059 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:50.185663 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:50.186644 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:50.187407 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:50.187511 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:50.190538 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:50.190542 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:50.190542 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:50.190555 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:50.193236 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:50.193710 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:50.193714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:50.193714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:50.193877 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:50.194163 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:50.194164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:50.194164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:50.194511 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:50.194664 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:50.194816 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 22:49:50.194978 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":109,"threads_started":1}
I20260928 22:49:50.195397 27165 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa38796cf78e4a8b9ef9c859e02922bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa38796cf78e4a8b9ef9c859e02922bf. Latest consensus state: current_term: 1 leader_uuid: "aa38796cf78e4a8b9ef9c859e02922bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa38796cf78e4a8b9ef9c859e02922bf" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 43911 } } }
I20260928 22:49:50.195485 27165 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa38796cf78e4a8b9ef9c859e02922bf [sys.catalog]: This master's current role is: LEADER
I20260928 22:49:50.195499 27166 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa38796cf78e4a8b9ef9c859e02922bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa38796cf78e4a8b9ef9c859e02922bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa38796cf78e4a8b9ef9c859e02922bf" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 43911 } } }
I20260928 22:49:50.195581 27166 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa38796cf78e4a8b9ef9c859e02922bf [sys.catalog]: This master's current role is: LEADER
I20260928 22:49:50.195760 27179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:49:50.196338 27179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:49:50.198175 27179 catalog_manager.cc:1347] Generated new cluster ID: 7582e63e1c814632b0bf31c65896e43e
I20260928 22:49:50.198230 27179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:49:50.218647 27179 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:49:50.219357 27179 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:49:50.221352 27179 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa38796cf78e4a8b9ef9c859e02922bf: Generated new TSK 0
I20260928 22:49:50.221550 27179 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.121.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab.
Entry for principal kudu/127.16.121.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab.
I20260928 22:49:50.250684 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.129:0
--local_ip_for_outbound_sockets=127.16.121.129
--webserver_interface=127.16.121.129
--webserver_port=0
--tserver_master_addrs=127.16.121.190:43911
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab
--principal=kudu/127.16.121.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:36469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:49:50.335829 27185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:50.336040 27185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:50.336066 27185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:50.337419 27185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:50.337474 27185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:50.337502 27185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.129
I20260928 22:49:50.339097 27185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:36469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab
--principal=kudu/127.16.121.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.121.129
--webserver_port=0
--tserver_master_addrs=127.16.121.190:43911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.129
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:50.339404 27185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:50.339681 27185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:50.345280 27191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:50.345291 27185 server_base.cc:1034] running on GCE node
W20260928 22:49:50.345261 27190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:50.345255 27193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:50.345769 27185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:50.346084 27185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:50.347303 27185 hybrid_clock.cc:648] HybridClock initialized: now 1790635790347335 us; error 106 us; skew 500 ppm
Sep 28 22:49:50 dist-test-slave-56vw krb5kdc[27078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635790, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:50.350642 27185 init.cc:377] Logged in from keytab as kudu/127.16.121.129@KRBTEST.COM (short username kudu)
I20260928 22:49:50.351228 27185 webserver.cc:460] Webserver started at http://127.16.121.129:39209/ using document root <none> and password file <none>
I20260928 22:49:50.351507 27185 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:50.351583 27185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:50.351740 27185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:50.352880 27185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/data/instance:
uuid: "35b6a83b479d446f85708535f2ba394a"
format_stamp: "Formatted at 2026-09-28 22:49:50 on dist-test-slave-56vw"
server_key: "e97e536de0078fe812d37b78689bb0b1"
server_key_iv: "2dc93c686f58235aac3428abdc9cfc86"
server_key_version: "encryptionkey@0"
I20260928 22:49:50.353271 27185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance:
uuid: "35b6a83b479d446f85708535f2ba394a"
format_stamp: "Formatted at 2026-09-28 22:49:50 on dist-test-slave-56vw"
server_key: "e97e536de0078fe812d37b78689bb0b1"
server_key_iv: "2dc93c686f58235aac3428abdc9cfc86"
server_key_version: "encryptionkey@0"
I20260928 22:49:50.354740 27185 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:49:50.355670 27200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:50.355860 27185 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 22:49:50.355971 27185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/wal
uuid: "35b6a83b479d446f85708535f2ba394a"
format_stamp: "Formatted at 2026-09-28 22:49:50 on dist-test-slave-56vw"
server_key: "e97e536de0078fe812d37b78689bb0b1"
server_key_iv: "2dc93c686f58235aac3428abdc9cfc86"
server_key_version: "encryptionkey@0"
I20260928 22:49:50.356074 27185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:50.370323 27185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:50.372905 27185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:50.373095 27185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:50.373347 27185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:49:50.373723 27185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:49:50.373780 27185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:50.373821 27185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:49:50.373847 27185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:50.380205 27185 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.129:44949
I20260928 22:49:50.380292 27313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.129:44949 every 8 connection(s)
I20260928 22:49:50.380743 27185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
I20260928 22:49:50.386636 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 27185
I20260928 22:49:50.386750 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance
I20260928 22:49:50.386991 16870 external_mini_cluster.cc:1428] Setting key c3547947ca2da5c238f9515242b19a9b
Sep 28 22:49:50 dist-test-slave-56vw krb5kdc[27078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635790, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:49:50.391671 27162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:50.381713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:56999 (local address 127.16.121.190:43911)
0928 22:49:50.381801 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:50.381804 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:50.382018 (+ 214us) server_negotiation.cc:407] Connection header received
0928 22:49:50.382661 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:50.382663 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:50.382675 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:50.382741 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:50.383876 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:50.384409 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:50.385560 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:50.385765 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:50.385829 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:50.388588 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:50.388590 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:50.388590 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:50.388599 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:50.390053 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:50.390568 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:50.390569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:50.390569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:50.390626 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:50.390920 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:50.390921 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:50.390921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:50.391031 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:50.391152 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:50.391490 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 22:49:50.391592 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 22:49:50.391955 27316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:50.381654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:43911 (local address 127.16.121.129:56999)
0928 22:49:50.381874 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:50.381918 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:50.382544 (+ 626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:50.382853 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:50.382858 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:50.383548 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:50.383756 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:50.383762 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:50.384527 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:50.384528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:50.385465 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:50.385468 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:50.385860 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:50.385861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:50.385980 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:50.386638 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:50.386647 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:50.388468 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:50.390135 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:50.390141 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:50.390144 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:50.390491 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:50.390697 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:50.390698 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:50.390698 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:50.390848 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:50.391148 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:50.391154 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:50.391347 (+ 193us) client_negotiation.cc:769] Sending connection context
0928 22:49:50.391471 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 22:49:50.391664 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":114,"threads_started":1}
I20260928 22:49:50.392555 27314 heartbeater.cc:347] Connected to a master server at 127.16.121.190:43911
I20260928 22:49:50.392670 27314 heartbeater.cc:464] Registering TS with master...
I20260928 22:49:50.392891 27314 heartbeater.cc:511] Master 127.16.121.190:43911 requested a full tablet report, sending...
I20260928 22:49:50.393514 27123 ts_manager.cc:194] Registered new tserver with Master: 35b6a83b479d446f85708535f2ba394a (127.16.121.129:44949)
I20260928 22:49:50.394277 27123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.129@KRBTEST.COM'} at 127.16.121.129:56999
WARNING: no policy specified for kudu/127.16.121.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab.
Entry for principal kudu/127.16.121.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab.
I20260928 22:49:50.419225 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.130:0
--local_ip_for_outbound_sockets=127.16.121.130
--webserver_interface=127.16.121.130
--webserver_port=0
--tserver_master_addrs=127.16.121.190:43911
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab
--principal=kudu/127.16.121.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:36469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:49:50.503942 27319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:50.504168 27319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:50.504192 27319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:50.505537 27319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:50.505641 27319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:50.505694 27319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.130
I20260928 22:49:50.507288 27319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:36469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab
--principal=kudu/127.16.121.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.121.130
--webserver_port=0
--tserver_master_addrs=127.16.121.190:43911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.130
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:50.507555 27319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:50.507843 27319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:50.513218 27324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:50.513231 27327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:50.513234 27325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:50.513525 27319 server_base.cc:1034] running on GCE node
I20260928 22:49:50.513705 27319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:50.513926 27319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:50.515139 27319 hybrid_clock.cc:648] HybridClock initialized: now 1790635790515107 us; error 40 us; skew 500 ppm
Sep 28 22:49:50 dist-test-slave-56vw krb5kdc[27078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635790, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:50.518265 27319 init.cc:377] Logged in from keytab as kudu/127.16.121.130@KRBTEST.COM (short username kudu)
I20260928 22:49:50.518851 27319 webserver.cc:460] Webserver started at http://127.16.121.130:38563/ using document root <none> and password file <none>
I20260928 22:49:50.519161 27319 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:50.519239 27319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:50.519415 27319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:50.520674 27319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/data/instance:
uuid: "40094175aae54c7c850737a0be41d621"
format_stamp: "Formatted at 2026-09-28 22:49:50 on dist-test-slave-56vw"
server_key: "a32d74df60cdd9664a4d35fee06b9db4"
server_key_iv: "4bca9a71253304c571e027974d467f96"
server_key_version: "encryptionkey@0"
I20260928 22:49:50.521095 27319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance:
uuid: "40094175aae54c7c850737a0be41d621"
format_stamp: "Formatted at 2026-09-28 22:49:50 on dist-test-slave-56vw"
server_key: "a32d74df60cdd9664a4d35fee06b9db4"
server_key_iv: "4bca9a71253304c571e027974d467f96"
server_key_version: "encryptionkey@0"
I20260928 22:49:50.522647 27319 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:49:50.523682 27334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:50.523984 27319 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:49:50.524108 27319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/wal
uuid: "40094175aae54c7c850737a0be41d621"
format_stamp: "Formatted at 2026-09-28 22:49:50 on dist-test-slave-56vw"
server_key: "a32d74df60cdd9664a4d35fee06b9db4"
server_key_iv: "4bca9a71253304c571e027974d467f96"
server_key_version: "encryptionkey@0"
I20260928 22:49:50.524189 27319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:50.535122 27319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:50.537519 27319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:50.537693 27319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:50.537986 27319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:49:50.538300 27319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:49:50.538353 27319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:50.538396 27319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:49:50.538427 27319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:50.544893 27319 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.130:33179
I20260928 22:49:50.544929 27447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.130:33179 every 8 connection(s)
I20260928 22:49:50.545274 27319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
Sep 28 22:49:50 dist-test-slave-56vw krb5kdc[27078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635790, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.130@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:49:50.554677 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 27319
I20260928 22:49:50.554840 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance
I20260928 22:49:50.555106 16870 external_mini_cluster.cc:1428] Setting key 89075ef54ae7f34c60671fd4ca41b79e
I20260928 22:49:50.555444 27162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:50.546039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:36807 (local address 127.16.121.190:43911)
0928 22:49:50.546100 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:50.546102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:50.546452 (+ 350us) server_negotiation.cc:407] Connection header received
0928 22:49:50.546913 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:50.546915 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:50.546927 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:50.547011 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:50.548018 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:50.548516 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:50.549582 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:50.549774 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:50.549834 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:50.552296 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:50.552297 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:50.552297 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:50.552305 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:50.553628 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:50.554078 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:50.554079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:50.554080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:50.554145 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:50.554495 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:50.554496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:50.554496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:50.554662 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:50.554745 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:50.555161 (+ 416us) server_negotiation.cc:299] Negotiation successful
0928 22:49:50.555334 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 22:49:50.555666 27450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:50.546022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:43911 (local address 127.16.121.130:36807)
0928 22:49:50.546177 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:50.546215 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:50.546803 (+ 588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:50.547122 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:50.547127 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:50.547724 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:50.547922 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:50.547927 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:50.548603 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:50.548604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:50.549501 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:50.549503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:50.549862 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:50.549863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:50.549970 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:50.550464 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:50.550472 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:50.552156 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:50.553778 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:50.553783 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:50.553785 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:50.554009 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:50.554281 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:50.554282 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:50.554282 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:50.554381 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:50.554757 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:50.554763 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:50.554983 (+ 220us) client_negotiation.cc:769] Sending connection context
0928 22:49:50.555132 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 22:49:50.555412 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":91,"threads_started":1}
I20260928 22:49:50.556159 27448 heartbeater.cc:347] Connected to a master server at 127.16.121.190:43911
I20260928 22:49:50.556269 27448 heartbeater.cc:464] Registering TS with master...
I20260928 22:49:50.556471 27448 heartbeater.cc:511] Master 127.16.121.190:43911 requested a full tablet report, sending...
I20260928 22:49:50.556857 27123 ts_manager.cc:194] Registered new tserver with Master: 40094175aae54c7c850737a0be41d621 (127.16.121.130:33179)
I20260928 22:49:50.557384 27123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.130@KRBTEST.COM'} at 127.16.121.130:36807
WARNING: no policy specified for kudu/127.16.121.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab.
Entry for principal kudu/127.16.121.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab.
I20260928 22:49:50.587922 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.131:0
--local_ip_for_outbound_sockets=127.16.121.131
--webserver_interface=127.16.121.131
--webserver_port=0
--tserver_master_addrs=127.16.121.190:43911
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab
--principal=kudu/127.16.121.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:36469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:49:50.677428 27453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:49:50.677665 27453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:49:50.677691 27453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:49:50.678984 27453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:49:50.679035 27453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:49:50.679134 27453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.131
I20260928 22:49:50.680670 27453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:36469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab
--principal=kudu/127.16.121.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.121.131
--webserver_port=0
--tserver_master_addrs=127.16.121.190:43911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.131
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:49:50.680892 27453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:49:50.681083 27453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:49:50.686650 27461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:50.686655 27459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:49:50.686655 27458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:49:50.687018 27453 server_base.cc:1034] running on GCE node
I20260928 22:49:50.687263 27453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:49:50.687493 27453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:49:50.688635 27453 hybrid_clock.cc:648] HybridClock initialized: now 1790635790688623 us; error 31 us; skew 500 ppm
Sep 28 22:49:50 dist-test-slave-56vw krb5kdc[27078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635790, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:49:50.691424 27453 init.cc:377] Logged in from keytab as kudu/127.16.121.131@KRBTEST.COM (short username kudu)
I20260928 22:49:50.691891 27453 webserver.cc:460] Webserver started at http://127.16.121.131:46703/ using document root <none> and password file <none>
I20260928 22:49:50.692138 27453 fs_manager.cc:359] Metadata directory not provided
I20260928 22:49:50.692209 27453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:49:50.692384 27453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:49:50.693638 27453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/data/instance:
uuid: "2b173ae8e76049d1a58421c462dba527"
format_stamp: "Formatted at 2026-09-28 22:49:50 on dist-test-slave-56vw"
server_key: "a9d04a9eada78e25ce1c149777f42fa3"
server_key_iv: "05672a8a600824af7a88dbff56c2785c"
server_key_version: "encryptionkey@0"
I20260928 22:49:50.694103 27453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance:
uuid: "2b173ae8e76049d1a58421c462dba527"
format_stamp: "Formatted at 2026-09-28 22:49:50 on dist-test-slave-56vw"
server_key: "a9d04a9eada78e25ce1c149777f42fa3"
server_key_iv: "05672a8a600824af7a88dbff56c2785c"
server_key_version: "encryptionkey@0"
I20260928 22:49:50.695788 27453 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:49:50.696799 27468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:50.697060 27453 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:49:50.697183 27453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/wal
uuid: "2b173ae8e76049d1a58421c462dba527"
format_stamp: "Formatted at 2026-09-28 22:49:50 on dist-test-slave-56vw"
server_key: "a9d04a9eada78e25ce1c149777f42fa3"
server_key_iv: "05672a8a600824af7a88dbff56c2785c"
server_key_version: "encryptionkey@0"
I20260928 22:49:50.697382 27453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:49:50.707975 27453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:49:50.710326 27453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:49:50.710567 27453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:49:50.710839 27453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:49:50.711256 27453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:49:50.711313 27453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:50.711354 27453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:49:50.711382 27453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:49:50.717589 27453 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.131:44449
I20260928 22:49:50.717643 27581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.131:44449 every 8 connection(s)
I20260928 22:49:50.717975 27453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
I20260928 22:49:50.723353 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 27453
I20260928 22:49:50.723487 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance
I20260928 22:49:50.723662 16870 external_mini_cluster.cc:1428] Setting key 83fa60b4878da40fe4363ebd5dde0589
Sep 28 22:49:50 dist-test-slave-56vw krb5kdc[27078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635790, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.131@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:49:50.727902 27162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:50.718825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:35341 (local address 127.16.121.190:43911)
0928 22:49:50.718944 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:50.718947 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:50.719138 (+ 191us) server_negotiation.cc:407] Connection header received
0928 22:49:50.719730 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:50.719731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:50.719744 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:50.719795 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:50.720806 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:50.721316 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:50.722421 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:50.722618 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:50.722706 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:50.725239 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:50.725240 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:50.725241 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:50.725248 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:50.726562 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:50.726956 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:50.726957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:50.726958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:50.727017 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:50.727348 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:50.727348 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:50.727349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:50.727468 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:50.727539 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:50.727771 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 22:49:50.727837 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 22:49:50.728195 27584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:50.718740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:43911 (local address 127.16.121.131:35341)
0928 22:49:50.718979 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:50.719035 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:50.719608 (+ 573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:50.719900 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:50.719904 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:50.720494 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:49:50.720693 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:50.720698 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:50.721478 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:50.721479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:50.722326 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:50.722328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:50.722718 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:50.722719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:50.722853 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:50.723468 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:49:50.723476 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:49:50.725134 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:49:50.726642 (+ 1508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:50.726648 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:50.726650 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:50.726869 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:50.727163 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:49:50.727164 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:49:50.727165 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:49:50.727284 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:49:50.727530 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:49:50.727534 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:49:50.727653 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 22:49:50.727738 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 22:49:50.727884 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":85,"threads_started":1}
I20260928 22:49:50.728688 27582 heartbeater.cc:347] Connected to a master server at 127.16.121.190:43911
I20260928 22:49:50.728791 27582 heartbeater.cc:464] Registering TS with master...
I20260928 22:49:50.728971 27582 heartbeater.cc:511] Master 127.16.121.190:43911 requested a full tablet report, sending...
I20260928 22:49:50.729445 27123 ts_manager.cc:194] Registered new tserver with Master: 2b173ae8e76049d1a58421c462dba527 (127.16.121.131:44449)
I20260928 22:49:50.729815 27123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.131@KRBTEST.COM'} at 127.16.121.131:35341
I20260928 22:49:50.735672 16870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:49:50.744490 27162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:50.737091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56600 (local address 127.16.121.190:43911)
0928 22:49:50.737244 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:50.737247 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:50.737419 (+ 172us) server_negotiation.cc:407] Connection header received
0928 22:49:50.737504 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:50.737505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:50.737516 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:50.737574 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:49:50.738199 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:50.738704 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:50.739542 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:50.739742 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:50.739836 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:50.741326 (+ 1490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:49:50.741328 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:49:50.741329 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:49:50.741340 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:49:50.743002 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:50.743600 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:50.743601 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:50.743601 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:50.743666 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:49:50.743872 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:49:50.743873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:49:50.743874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:49:50.744041 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:49:50.744147 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:50.744291 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 22:49:50.744382 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260928 22:49:50.746817 27123 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:49:50.749042 27123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:49:50.756453 27595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:50.752084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36460 (local address 127.16.121.129:44949)
0928 22:49:50.752281 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:50.752287 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:50.752437 (+ 150us) server_negotiation.cc:407] Connection header received
0928 22:49:50.752544 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:50.752546 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:50.752564 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:50.752645 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:49:50.752915 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:50.753815 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:50.755237 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:50.756004 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:50.756078 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:50.756141 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:50.756302 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 22:49:50.756345 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":91,"threads_started":1}
I20260928 22:49:50.756440 27593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:50.751997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:44949 (local address 127.0.0.1:36460)
0928 22:49:50.752285 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:50.752337 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:50.752441 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:50.752746 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:50.752749 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:50.752751 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:49:50.752829 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:50.752831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:50.753925 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:50.753926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:50.755142 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:50.755145 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:50.756077 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:50.756077 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:50.756157 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:50.756172 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 22:49:50.756283 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 22:49:50.756346 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":102,"threads_started":1}
I20260928 22:49:50.756546 27596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:50.752101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33964 (local address 127.16.121.130:33179)
0928 22:49:50.752288 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:50.752292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:50.752443 (+ 151us) server_negotiation.cc:407] Connection header received
0928 22:49:50.752651 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:50.752653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:50.752669 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:50.752720 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:49:50.753030 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:50.754317 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:50.755542 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:50.756098 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:50.756172 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:50.756223 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:50.756418 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 22:49:50.756460 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":76,"threads_started":1}
I20260928 22:49:50.757046 27597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:50.751999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.130:33179 (local address 127.0.0.1:33964)
0928 22:49:50.752353 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:50.752368 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:50.752472 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:50.752815 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:50.752816 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:50.752818 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:49:50.752849 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:50.752851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:50.754447 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:50.754448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:50.755453 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:50.755455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:50.756260 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:50.756261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:50.756328 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:50.756332 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:49:50.756886 (+ 554us) client_negotiation.cc:240] Negotiation successful
0928 22:49:50.756948 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":11,"thread_start_us":67,"threads_started":1}
I20260928 22:49:50.757426 27594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:50.752240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.131:44449 (local address 127.0.0.1:38938)
0928 22:49:50.752835 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:50.752846 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:50.752900 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:50.753159 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:50.753160 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:50.753162 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:49:50.753203 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:50.753204 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:50.754558 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:50.754560 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:50.755922 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:50.755925 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:50.757037 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:50.757038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:50.757242 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:50.757247 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:49:50.757308 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 22:49:50.757348 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":163,"threads_started":1}
I20260928 22:49:50.757493 27598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:50.752564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38938 (local address 127.16.121.131:44449)
0928 22:49:50.752781 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:50.752786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:50.752967 (+ 181us) server_negotiation.cc:407] Connection header received
0928 22:49:50.753040 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:50.753042 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:50.753060 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:50.753149 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:49:50.753268 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:50.754436 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:50.756025 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:50.756964 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:50.757039 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:50.757090 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:50.757341 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 22:49:50.757389 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":97,"threads_started":1}
I20260928 22:49:50.757378 27248 tablet_service.cc:1467] Processing CreateTablet for tablet 03c81b9a3a5642eabbef1e82f8eedfef (DEFAULT_TABLE table=test-table [id=672ed0afeff3461797f406a41af10cc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:49:50.757792 27248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03c81b9a3a5642eabbef1e82f8eedfef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:50.757784 27382 tablet_service.cc:1467] Processing CreateTablet for tablet 03c81b9a3a5642eabbef1e82f8eedfef (DEFAULT_TABLE table=test-table [id=672ed0afeff3461797f406a41af10cc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:49:50.758118 27382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03c81b9a3a5642eabbef1e82f8eedfef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:50.758387 27516 tablet_service.cc:1467] Processing CreateTablet for tablet 03c81b9a3a5642eabbef1e82f8eedfef (DEFAULT_TABLE table=test-table [id=672ed0afeff3461797f406a41af10cc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:49:50.758711 27516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03c81b9a3a5642eabbef1e82f8eedfef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:49:50.762048 27601 tablet_bootstrap.cc:492] T 03c81b9a3a5642eabbef1e82f8eedfef P 2b173ae8e76049d1a58421c462dba527: Bootstrap starting.
I20260928 22:49:50.762331 27600 tablet_bootstrap.cc:492] T 03c81b9a3a5642eabbef1e82f8eedfef P 40094175aae54c7c850737a0be41d621: Bootstrap starting.
I20260928 22:49:50.762373 27599 tablet_bootstrap.cc:492] T 03c81b9a3a5642eabbef1e82f8eedfef P 35b6a83b479d446f85708535f2ba394a: Bootstrap starting.
I20260928 22:49:50.762766 27601 tablet_bootstrap.cc:654] T 03c81b9a3a5642eabbef1e82f8eedfef P 2b173ae8e76049d1a58421c462dba527: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:50.762965 27599 tablet_bootstrap.cc:654] T 03c81b9a3a5642eabbef1e82f8eedfef P 35b6a83b479d446f85708535f2ba394a: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:50.762969 27600 tablet_bootstrap.cc:654] T 03c81b9a3a5642eabbef1e82f8eedfef P 40094175aae54c7c850737a0be41d621: Neither blocks nor log segments found. Creating new log.
I20260928 22:49:50.763427 27601 log.cc:824] T 03c81b9a3a5642eabbef1e82f8eedfef P 2b173ae8e76049d1a58421c462dba527: Log is configured to *not* fsync() on all Append() calls
I20260928 22:49:50.763475 27599 log.cc:824] T 03c81b9a3a5642eabbef1e82f8eedfef P 35b6a83b479d446f85708535f2ba394a: Log is configured to *not* fsync() on all Append() calls
I20260928 22:49:50.763558 27600 log.cc:824] T 03c81b9a3a5642eabbef1e82f8eedfef P 40094175aae54c7c850737a0be41d621: Log is configured to *not* fsync() on all Append() calls
I20260928 22:49:50.764221 27601 tablet_bootstrap.cc:492] T 03c81b9a3a5642eabbef1e82f8eedfef P 2b173ae8e76049d1a58421c462dba527: No bootstrap required, opened a new log
I20260928 22:49:50.764297 27601 ts_tablet_manager.cc:1397] T 03c81b9a3a5642eabbef1e82f8eedfef P 2b173ae8e76049d1a58421c462dba527: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:49:50.764298 27600 tablet_bootstrap.cc:492] T 03c81b9a3a5642eabbef1e82f8eedfef P 40094175aae54c7c850737a0be41d621: No bootstrap required, opened a new log
I20260928 22:49:50.764361 27600 ts_tablet_manager.cc:1397] T 03c81b9a3a5642eabbef1e82f8eedfef P 40094175aae54c7c850737a0be41d621: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:49:50.764679 27599 tablet_bootstrap.cc:492] T 03c81b9a3a5642eabbef1e82f8eedfef P 35b6a83b479d446f85708535f2ba394a: No bootstrap required, opened a new log
I20260928 22:49:50.764796 27599 ts_tablet_manager.cc:1397] T 03c81b9a3a5642eabbef1e82f8eedfef P 35b6a83b479d446f85708535f2ba394a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:49:50.765592 27601 raft_consensus.cc:348] T 03c81b9a3a5642eabbef1e82f8eedfef P 2b173ae8e76049d1a58421c462dba527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b6a83b479d446f85708535f2ba394a" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 44949 } } peers { permanent_uuid: "2b173ae8e76049d1a58421c462dba527" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 44449 } } peers { permanent_uuid: "40094175aae54c7c850737a0be41d621" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 33179 } }
I20260928 22:49:50.765722 27601 raft_consensus.cc:374] T 03c81b9a3a5642eabbef1e82f8eedfef P 2b173ae8e76049d1a58421c462dba527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:50.765743 27601 raft_consensus.cc:729] T 03c81b9a3a5642eabbef1e82f8eedfef P 2b173ae8e76049d1a58421c462dba527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b173ae8e76049d1a58421c462dba527, State: Initialized, Role: FOLLOWER
I20260928 22:49:50.765869 27601 consensus_queue.cc:260] T 03c81b9a3a5642eabbef1e82f8eedfef P 2b173ae8e76049d1a58421c462dba527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b6a83b479d446f85708535f2ba394a" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 44949 } } peers { permanent_uuid: "2b173ae8e76049d1a58421c462dba527" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 44449 } } peers { permanent_uuid: "40094175aae54c7c850737a0be41d621" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 33179 } }
I20260928 22:49:50.766168 27601 ts_tablet_manager.cc:1428] T 03c81b9a3a5642eabbef1e82f8eedfef P 2b173ae8e76049d1a58421c462dba527: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:49:50.766175 27582 heartbeater.cc:503] Master 127.16.121.190:43911 was elected leader, sending a full tablet report...
I20260928 22:49:50.766428 27600 raft_consensus.cc:348] T 03c81b9a3a5642eabbef1e82f8eedfef P 40094175aae54c7c850737a0be41d621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b6a83b479d446f85708535f2ba394a" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 44949 } } peers { permanent_uuid: "2b173ae8e76049d1a58421c462dba527" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 44449 } } peers { permanent_uuid: "40094175aae54c7c850737a0be41d621" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 33179 } }
I20260928 22:49:50.766604 27600 raft_consensus.cc:374] T 03c81b9a3a5642eabbef1e82f8eedfef P 40094175aae54c7c850737a0be41d621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:50.766647 27600 raft_consensus.cc:729] T 03c81b9a3a5642eabbef1e82f8eedfef P 40094175aae54c7c850737a0be41d621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40094175aae54c7c850737a0be41d621, State: Initialized, Role: FOLLOWER
I20260928 22:49:50.766774 27600 consensus_queue.cc:260] T 03c81b9a3a5642eabbef1e82f8eedfef P 40094175aae54c7c850737a0be41d621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b6a83b479d446f85708535f2ba394a" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 44949 } } peers { permanent_uuid: "2b173ae8e76049d1a58421c462dba527" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 44449 } } peers { permanent_uuid: "40094175aae54c7c850737a0be41d621" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 33179 } }
I20260928 22:49:50.767124 27600 ts_tablet_manager.cc:1428] T 03c81b9a3a5642eabbef1e82f8eedfef P 40094175aae54c7c850737a0be41d621: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:49:50.767211 27448 heartbeater.cc:503] Master 127.16.121.190:43911 was elected leader, sending a full tablet report...
I20260928 22:49:50.767845 27599 raft_consensus.cc:348] T 03c81b9a3a5642eabbef1e82f8eedfef P 35b6a83b479d446f85708535f2ba394a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b6a83b479d446f85708535f2ba394a" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 44949 } } peers { permanent_uuid: "2b173ae8e76049d1a58421c462dba527" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 44449 } } peers { permanent_uuid: "40094175aae54c7c850737a0be41d621" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 33179 } }
I20260928 22:49:50.768040 27599 raft_consensus.cc:374] T 03c81b9a3a5642eabbef1e82f8eedfef P 35b6a83b479d446f85708535f2ba394a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:49:50.768080 27599 raft_consensus.cc:729] T 03c81b9a3a5642eabbef1e82f8eedfef P 35b6a83b479d446f85708535f2ba394a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35b6a83b479d446f85708535f2ba394a, State: Initialized, Role: FOLLOWER
I20260928 22:49:50.768193 27599 consensus_queue.cc:260] T 03c81b9a3a5642eabbef1e82f8eedfef P 35b6a83b479d446f85708535f2ba394a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b6a83b479d446f85708535f2ba394a" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 44949 } } peers { permanent_uuid: "2b173ae8e76049d1a58421c462dba527" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 44449 } } peers { permanent_uuid: "40094175aae54c7c850737a0be41d621" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 33179 } }
I20260928 22:49:50.768486 27599 ts_tablet_manager.cc:1428] T 03c81b9a3a5642eabbef1e82f8eedfef P 35b6a83b479d446f85708535f2ba394a: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 22:49:50.768524 27314 heartbeater.cc:503] Master 127.16.121.190:43911 was elected leader, sending a full tablet report...
W20260928 22:49:50.795910 27449 tablet.cc:2369] T 03c81b9a3a5642eabbef1e82f8eedfef P 40094175aae54c7c850737a0be41d621: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:49:50.819329 27606 raft_consensus.cc:482] T 03c81b9a3a5642eabbef1e82f8eedfef P 40094175aae54c7c850737a0be41d621 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:49:50.819521 27606 raft_consensus.cc:504] T 03c81b9a3a5642eabbef1e82f8eedfef P 40094175aae54c7c850737a0be41d621 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b6a83b479d446f85708535f2ba394a" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 44949 } } peers { permanent_uuid: "2b173ae8e76049d1a58421c462dba527" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 44449 } } peers { permanent_uuid: "40094175aae54c7c850737a0be41d621" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 33179 } }
I20260928 22:49:50.819941 27606 leader_election.cc:288] T 03c81b9a3a5642eabbef1e82f8eedfef P 40094175aae54c7c850737a0be41d621 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35b6a83b479d446f85708535f2ba394a (127.16.121.129:44949), 2b173ae8e76049d1a58421c462dba527 (127.16.121.131:44449)
I20260928 22:49:50.822961 27595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:50.820282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:36261 (local address 127.16.121.129:44949)
0928 22:49:50.820386 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:50.820389 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:50.820397 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:49:50.820429 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:50.820430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:50.820441 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:50.820498 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:49:50.820657 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:50.821149 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:50.822194 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:50.822581 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:50.822666 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:50.822675 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:50.822856 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 22:49:50.822905 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 22:49:50.822959 27450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:50.820150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:44949 (local address 127.16.121.130:36261)
0928 22:49:50.820221 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:50.820236 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:50.820320 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:50.820501 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:50.820502 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:50.820504 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:49:50.820584 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:50.820585 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:50.821237 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:50.821238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:50.822084 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:50.822086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:50.822673 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:50.822674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:50.822755 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:50.822758 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:49:50.822838 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 22:49:50.822881 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":26}
I20260928 22:49:50.823081 27598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:50.820379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:35069 (local address 127.16.121.131:44449)
0928 22:49:50.820438 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:50.820440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:50.820562 (+ 122us) server_negotiation.cc:407] Connection header received
0928 22:49:50.820630 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:50.820631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:50.820641 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:50.820705 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:49:50.820859 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:50.821330 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:50.822283 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:50.822704 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:50.822794 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:50.822802 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:50.822954 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 22:49:50.822989 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 22:49:50.823097 27608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:50.820286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.131:44449 (local address 127.16.121.130:35069)
0928 22:49:50.820481 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:50.820491 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:49:50.820571 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:49:50.820712 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:49:50.820712 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:49:50.820714 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:49:50.820788 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:50.820790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:50.821420 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:50.821421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:50.822170 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:49:50.822172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:49:50.822794 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:49:50.822795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:49:50.822870 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:50.822874 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:49:50.822936 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 22:49:50.822977 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":116,"threads_started":1}
I20260928 22:49:50.823431 27536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c81b9a3a5642eabbef1e82f8eedfef" candidate_uuid: "40094175aae54c7c850737a0be41d621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b173ae8e76049d1a58421c462dba527" is_pre_election: true
I20260928 22:49:50.823431 27268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c81b9a3a5642eabbef1e82f8eedfef" candidate_uuid: "40094175aae54c7c850737a0be41d621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35b6a83b479d446f85708535f2ba394a" is_pre_election: true
I20260928 22:49:50.823624 27268 raft_consensus.cc:2454] T 03c81b9a3a5642eabbef1e82f8eedfef P 35b6a83b479d446f85708535f2ba394a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40094175aae54c7c850737a0be41d621 in term 0.
I20260928 22:49:50.823623 27536 raft_consensus.cc:2454] T 03c81b9a3a5642eabbef1e82f8eedfef P 2b173ae8e76049d1a58421c462dba527 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40094175aae54c7c850737a0be41d621 in term 0.
I20260928 22:49:50.823866 27337 leader_election.cc:302] T 03c81b9a3a5642eabbef1e82f8eedfef P 40094175aae54c7c850737a0be41d621 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35b6a83b479d446f85708535f2ba394a, 40094175aae54c7c850737a0be41d621; no voters:
I20260928 22:49:50.824033 27606 raft_consensus.cc:2781] T 03c81b9a3a5642eabbef1e82f8eedfef P 40094175aae54c7c850737a0be41d621 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:49:50.824105 27606 raft_consensus.cc:482] T 03c81b9a3a5642eabbef1e82f8eedfef P 40094175aae54c7c850737a0be41d621 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:49:50.824123 27606 raft_consensus.cc:3037] T 03c81b9a3a5642eabbef1e82f8eedfef P 40094175aae54c7c850737a0be41d621 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:50.824952 27606 raft_consensus.cc:504] T 03c81b9a3a5642eabbef1e82f8eedfef P 40094175aae54c7c850737a0be41d621 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b6a83b479d446f85708535f2ba394a" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 44949 } } peers { permanent_uuid: "2b173ae8e76049d1a58421c462dba527" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 44449 } } peers { permanent_uuid: "40094175aae54c7c850737a0be41d621" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 33179 } }
I20260928 22:49:50.825067 27606 leader_election.cc:288] T 03c81b9a3a5642eabbef1e82f8eedfef P 40094175aae54c7c850737a0be41d621 [CANDIDATE]: Term 1 election: Requested vote from peers 35b6a83b479d446f85708535f2ba394a (127.16.121.129:44949), 2b173ae8e76049d1a58421c462dba527 (127.16.121.131:44449)
I20260928 22:49:50.825248 27268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c81b9a3a5642eabbef1e82f8eedfef" candidate_uuid: "40094175aae54c7c850737a0be41d621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35b6a83b479d446f85708535f2ba394a"
I20260928 22:49:50.825343 27268 raft_consensus.cc:3037] T 03c81b9a3a5642eabbef1e82f8eedfef P 35b6a83b479d446f85708535f2ba394a [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:50.825322 27536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c81b9a3a5642eabbef1e82f8eedfef" candidate_uuid: "40094175aae54c7c850737a0be41d621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b173ae8e76049d1a58421c462dba527"
I20260928 22:49:50.825438 27536 raft_consensus.cc:3037] T 03c81b9a3a5642eabbef1e82f8eedfef P 2b173ae8e76049d1a58421c462dba527 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:49:50.826146 27268 raft_consensus.cc:2454] T 03c81b9a3a5642eabbef1e82f8eedfef P 35b6a83b479d446f85708535f2ba394a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40094175aae54c7c850737a0be41d621 in term 1.
I20260928 22:49:50.826325 27536 raft_consensus.cc:2454] T 03c81b9a3a5642eabbef1e82f8eedfef P 2b173ae8e76049d1a58421c462dba527 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40094175aae54c7c850737a0be41d621 in term 1.
I20260928 22:49:50.826352 27337 leader_election.cc:302] T 03c81b9a3a5642eabbef1e82f8eedfef P 40094175aae54c7c850737a0be41d621 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35b6a83b479d446f85708535f2ba394a, 40094175aae54c7c850737a0be41d621; no voters:
I20260928 22:49:50.826436 27606 raft_consensus.cc:2781] T 03c81b9a3a5642eabbef1e82f8eedfef P 40094175aae54c7c850737a0be41d621 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:49:50.826594 27606 raft_consensus.cc:686] T 03c81b9a3a5642eabbef1e82f8eedfef P 40094175aae54c7c850737a0be41d621 [term 1 LEADER]: Becoming Leader. State: Replica: 40094175aae54c7c850737a0be41d621, State: Running, Role: LEADER
I20260928 22:49:50.826720 27606 consensus_queue.cc:237] T 03c81b9a3a5642eabbef1e82f8eedfef P 40094175aae54c7c850737a0be41d621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b6a83b479d446f85708535f2ba394a" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 44949 } } peers { permanent_uuid: "2b173ae8e76049d1a58421c462dba527" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 44449 } } peers { permanent_uuid: "40094175aae54c7c850737a0be41d621" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 33179 } }
I20260928 22:49:50.827654 27122 catalog_manager.cc:5495] T 03c81b9a3a5642eabbef1e82f8eedfef P 40094175aae54c7c850737a0be41d621 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40094175aae54c7c850737a0be41d621 (127.16.121.130). New cstate: current_term: 1 leader_uuid: "40094175aae54c7c850737a0be41d621" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b6a83b479d446f85708535f2ba394a" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 44949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b173ae8e76049d1a58421c462dba527" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 44449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40094175aae54c7c850737a0be41d621" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 33179 } health_report { overall_health: HEALTHY } } }
I20260928 22:49:50.851842 27596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:50.849108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33972 (local address 127.16.121.130:33179)
0928 22:49:50.849175 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 22:49:50.849177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:49:50.849185 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:49:50.849307 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:49:50.849307 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:49:50.849326 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:49:50.849393 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:49:50.849554 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:50.850036 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:50.850924 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:49:50.851195 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:49:50.851264 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:49:50.851460 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:49:50.851568 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:49:50.851646 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 22:49:50.851665 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 22:49:50.851686 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 22:49:50.853902 27536 raft_consensus.cc:1261] T 03c81b9a3a5642eabbef1e82f8eedfef P 2b173ae8e76049d1a58421c462dba527 [term 1 FOLLOWER]: Refusing update from remote peer 40094175aae54c7c850737a0be41d621: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:49:50.853901 27268 raft_consensus.cc:1261] T 03c81b9a3a5642eabbef1e82f8eedfef P 35b6a83b479d446f85708535f2ba394a [term 1 FOLLOWER]: Refusing update from remote peer 40094175aae54c7c850737a0be41d621: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:49:50.854302 27606 consensus_queue.cc:1035] T 03c81b9a3a5642eabbef1e82f8eedfef P 40094175aae54c7c850737a0be41d621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35b6a83b479d446f85708535f2ba394a" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 44949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:49:50.854490 27609 consensus_queue.cc:1035] T 03c81b9a3a5642eabbef1e82f8eedfef P 40094175aae54c7c850737a0be41d621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b173ae8e76049d1a58421c462dba527" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 44449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:49:50.858160 27614 mvcc.cc:205] Tried to move back new op lower bound from 7334444199334436864 to 7334444199226818560. Current Snapshot: MvccSnapshot[applied={T|T < 7334444199334436864}]
I20260928 22:49:50.859856 27123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:49:50.859954 27123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:49:50.860796 27123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa38796cf78e4a8b9ef9c859e02922bf: Sending DeleteTablet for 3 replicas of tablet 03c81b9a3a5642eabbef1e82f8eedfef
I20260928 22:49:50.861171 27382 tablet_service.cc:1514] Processing DeleteTablet for tablet 03c81b9a3a5642eabbef1e82f8eedfef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:49:50 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:33964
I20260928 22:49:50.861182 27516 tablet_service.cc:1514] Processing DeleteTablet for tablet 03c81b9a3a5642eabbef1e82f8eedfef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:49:50 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:38938
I20260928 22:49:50.861174 27248 tablet_service.cc:1514] Processing DeleteTablet for tablet 03c81b9a3a5642eabbef1e82f8eedfef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:49:50 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:36460
I20260928 22:49:50.861440 27623 tablet_replica.cc:330] T 03c81b9a3a5642eabbef1e82f8eedfef P 35b6a83b479d446f85708535f2ba394a: stopping tablet replica
I20260928 22:49:50.861553 27623 raft_consensus.cc:2229] T 03c81b9a3a5642eabbef1e82f8eedfef P 35b6a83b479d446f85708535f2ba394a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:49:50.861667 27622 tablet_replica.cc:330] T 03c81b9a3a5642eabbef1e82f8eedfef P 40094175aae54c7c850737a0be41d621: stopping tablet replica
I20260928 22:49:50.861764 27622 raft_consensus.cc:2229] T 03c81b9a3a5642eabbef1e82f8eedfef P 40094175aae54c7c850737a0be41d621 [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:49:50.861864 27622 raft_consensus.cc:2258] T 03c81b9a3a5642eabbef1e82f8eedfef P 40094175aae54c7c850737a0be41d621 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:49:50.861935 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 27185
I20260928 22:49:50.863093 27622 ts_tablet_manager.cc:1905] T 03c81b9a3a5642eabbef1e82f8eedfef P 40094175aae54c7c850737a0be41d621: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:49:50.864471 27621 tablet_replica.cc:330] T 03c81b9a3a5642eabbef1e82f8eedfef P 2b173ae8e76049d1a58421c462dba527: stopping tablet replica
I20260928 22:49:50.864554 27621 raft_consensus.cc:2229] T 03c81b9a3a5642eabbef1e82f8eedfef P 2b173ae8e76049d1a58421c462dba527 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:49:50.864634 27621 raft_consensus.cc:2258] T 03c81b9a3a5642eabbef1e82f8eedfef P 2b173ae8e76049d1a58421c462dba527 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:49:50.864771 27622 ts_tablet_manager.cc:1918] T 03c81b9a3a5642eabbef1e82f8eedfef P 40094175aae54c7c850737a0be41d621: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:49:50.864842 27622 log.cc:1196] T 03c81b9a3a5642eabbef1e82f8eedfef P 40094175aae54c7c850737a0be41d621: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-1/wal/wals/03c81b9a3a5642eabbef1e82f8eedfef
I20260928 22:49:50.865111 27622 ts_tablet_manager.cc:1939] T 03c81b9a3a5642eabbef1e82f8eedfef P 40094175aae54c7c850737a0be41d621: Deleting consensus metadata
I20260928 22:49:50.865137 27621 ts_tablet_manager.cc:1905] T 03c81b9a3a5642eabbef1e82f8eedfef P 2b173ae8e76049d1a58421c462dba527: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:49:50.865512 27107 catalog_manager.cc:4841] TS 40094175aae54c7c850737a0be41d621 (127.16.121.130:33179): tablet 03c81b9a3a5642eabbef1e82f8eedfef (table test-table [id=672ed0afeff3461797f406a41af10cc1]) successfully deleted
I20260928 22:49:50.866611 27621 ts_tablet_manager.cc:1918] T 03c81b9a3a5642eabbef1e82f8eedfef P 2b173ae8e76049d1a58421c462dba527: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:49:50.866668 27621 log.cc:1196] T 03c81b9a3a5642eabbef1e82f8eedfef P 2b173ae8e76049d1a58421c462dba527: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790635733329745-16870-0/minicluster-data/ts-2/wal/wals/03c81b9a3a5642eabbef1e82f8eedfef
I20260928 22:49:50.866871 27621 ts_tablet_manager.cc:1939] T 03c81b9a3a5642eabbef1e82f8eedfef P 2b173ae8e76049d1a58421c462dba527: Deleting consensus metadata
I20260928 22:49:50.867419 27109 catalog_manager.cc:4841] TS 2b173ae8e76049d1a58421c462dba527 (127.16.121.131:44449): tablet 03c81b9a3a5642eabbef1e82f8eedfef (table test-table [id=672ed0afeff3461797f406a41af10cc1]) successfully deleted
W20260928 22:49:50.869374 27109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.121.129:44949 (error 108)
I20260928 22:49:50.869531 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 27319
I20260928 22:49:50.870733 27594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:49:50.870594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:44949 (local address 127.0.0.1:36468)
0928 22:49:50.870656 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 22:49:50.870695 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.129:44949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 22:49:50.870985 27109 catalog_manager.cc:4584] TS 35b6a83b479d446f85708535f2ba394a (127.16.121.129:44949): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 03c81b9a3a5642eabbef1e82f8eedfef: Network error: Client connection negotiation failed: client connection to 127.16.121.129:44949: connect: Connection refused (error 111)
I20260928 22:49:50.876664 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 27453
I20260928 22:49:50.881515 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 27091
2026-09-28T22:49:50Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2033 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:49:50 dist-test-slave-56vw krb5kdc[27627](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:49:50 dist-test-slave-56vw krb5kdc[27627](info): set up 2 sockets
Sep 28 22:49:50 dist-test-slave-56vw krb5kdc[27627](info): commencing operation
krb5kdc: starting...
W20260928 22:49:51.940160 16870 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Sep 28 22:49:52 dist-test-slave-56vw krb5kdc[27627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:49:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:49:52Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.16.121.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.16.121.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.16.121.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/krb5kdc/rangeradmin_127.16.121.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.16.121.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/krb5kdc/rangeradmin_127.16.121.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.16.121.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.16.121.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.16.121.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/krb5kdc/rangerlookup_127.16.121.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.16.121.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/krb5kdc/rangerlookup_127.16.121.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.121.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.16.121.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.121.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/krb5kdc/HTTP_127.16.121.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.121.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/krb5kdc/HTTP_127.16.121.148@KRBTEST.COM.keytab.
I20260928 22:49:52.125659 16870 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskhFq4Cy/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/postgres -l logfile start
2026-09-28 22:49:55.028 UTC [27657] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 22:49:55.028 UTC [27657] LOG: listening on IPv4 address "127.16.121.148", port 37283
2026-09-28 22:49:55.033 UTC [27657] LOG: listening on Unix socket "/tmp/.s.PGSQL.37283"
2026-09-28 22:49:55.057 UTC [27660] LOG: database system was shut down at 2026-09-28 22:49:53 UTC
2026-09-28 22:49:55.062 UTC [27657] LOG: database system is ready to accept connections
I20260928 22:49:56.032013 16870 mini_postgres.cc:96] Postgres bound to 37283
2026-09-28 22:49:56.038 UTC [27668] FATAL: database "slave" does not exist
127.16.121.148:37283 - accepting connections
I20260928 22:49:56.039160 16870 mini_ranger.cc:161] Starting Ranger...
I20260928 22:49:56.054786 16870 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 22:49:56.683811 16870 mini_ranger.cc:87] Created ranger Postgres database
I20260928 22:49:56.683903 16870 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ranger-admin
2026-09-28 22:49:56,963 [I] DB FLAVOR :POSTGRES
2026-09-28 22:49:56,964 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 22:49:56,964 [I] Checking connection..
2026-09-28 22:49:56,964 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.121.148:37283/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 22:49:57,276 [I] Checking connection passed.
2026-09-28 22:49:57,277 [I] --------- Verifying version history table ---------
2026-09-28 22:49:57,277 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.121.148:37283/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 22:49:57,538 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 22:49:57,539 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.121.148:37283/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 22:49:57,798 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 22:49:57,798 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 22:49:57,798 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.121.148:37283/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 22:49:58.059 UTC [27732] WARNING: there is no transaction in progress
2026-09-28 22:49:58,067 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 22:49:58,067 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.121.148:37283/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 22:49:58,326 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 22:49:58,326 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 22:49:58,327 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.121.148:37283/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 22:49:58,603 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.121.148:37283/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 22:49:58,873 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.121.148:37283/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-56vw.c.gcp-upstream.internal','N') ;"
2026-09-28 22:49:59,134 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.121.148:37283/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 22:49:59,400 [I] Table x_portal_user does not exist in database ranger
2026-09-28 22:49:59,401 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.121.148:37283/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 22:49:59,663 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 22:49:59,663 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.121.148:37283/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 22:49:59,930 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.121.148:37283/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 22:50:00,178 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 22:50:00,178 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.121.148:37283/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 22:50:01.294 UTC [27840] WARNING: there is no transaction in progress
2026-09-28 22:50:01.316 UTC [27840] WARNING: there is no transaction in progress
2026-09-28 22:50:01.336 UTC [27840] WARNING: there is no transaction in progress
2026-09-28 22:50:01.356 UTC [27840] WARNING: there is no transaction in progress
2026-09-28 22:50:01.374 UTC [27840] WARNING: there is no transaction in progress
2026-09-28 22:50:01.397 UTC [27840] WARNING: there is no transaction in progress
2026-09-28 22:50:01.461 UTC [27840] WARNING: there is no transaction in progress
2026-09-28 22:50:01.469 UTC [27840] WARNING: there is no transaction in progress
2026-09-28 22:50:01.487 UTC [27840] WARNING: there is no transaction in progress
2026-09-28 22:50:01.499 UTC [27840] WARNING: there is no transaction in progress
2026-09-28 22:50:01.511 UTC [27840] WARNING: there is no transaction in progress
2026-09-28 22:50:01.528 UTC [27840] WARNING: there is no transaction in progress
2026-09-28 22:50:01.539 UTC [27840] WARNING: there is no transaction in progress
2026-09-28 22:50:01.545 UTC [27840] WARNING: there is no transaction in progress
2026-09-28 22:50:01.553 UTC [27840] WARNING: there is no transaction in progress
2026-09-28 22:50:01.562 UTC [27840] WARNING: there is no transaction in progress
2026-09-28 22:50:02,537 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 22:50:02,538 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.121.148:37283/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 22:50:02,799 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 22:50:02,799 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.121.148:37283/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 22:50:03,044 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 22:50:03,044 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.121.148:37283/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 22:50:03,318 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 22:50:03,318 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.121.148:37283/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 22:50:03,596 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.121.148:37283/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 22:50:03,850 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.121.148:37283/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-56vw.c.gcp-upstream.internal';"
2026-09-28 22:50:04,102 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 22:50:04,103 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.121.148:37283/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 22:50:04,393 [I] DB_PATCHES have already been applied
I20260928 22:50:04.399346 16870 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ranger-admin:/tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskhFq4Cy/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 22:50:04.399425 16870 mini_ranger.cc:193] Using host: 127.16.121.148
I20260928 22:50:04.402743 16870 mini_ranger.cc:227] Ranger admin URL: http://127.16.121.148:33159
Sep 28, 2026 10:50:04 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 10:50:04 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 10:50:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 10:50:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 10:50:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 10:50:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 10:50:05 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 22:50:05.564692 16870 mini_ranger.cc:160] Time spent starting Ranger: real 9.526s user 0.000s sys 0.010s
Sep 28, 2026 10:50:26 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 10:50:26 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 28, 2026 10:50:26 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 10:50:26 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 10:50:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 10:50:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 10:50:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 10:50:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 10:50:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 10:50:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 10:50:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 10:50:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 10:50:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 10:50:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 10:50:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 10:50:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 10:50:26 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 10:50:27 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 22:50:27.900188 16870 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.16.121.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.16.121.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.16.121.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/krb5kdc/rangerkms_127.16.121.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.16.121.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/krb5kdc/rangerkms_127.16.121.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.121.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.16.121.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.121.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/krb5kdc/HTTP_127.16.121.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.121.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/krb5kdc/HTTP_127.16.121.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 22:50:27 dist-test-slave-56vw krb5kdc[27627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635827, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 22:50:28.000114 16870 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 22:50:28.011354 16870 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 22:50:29.211586 16870 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 22:50:29.211656 16870 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ranger-kms
I20260928 22:50:29.211669 16870 mini_ranger_kms.cc:227] Using postgres at 127.16.121.148:37283
2026-09-28 22:50:29,383 [I] DB FLAVOR :POSTGRES
2026-09-28 22:50:29,384 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 22:50:29,384 [I] Checking connection
2026-09-28 22:50:29,384 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.121.148:37283/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 22:50:29,641 [I] connection success
2026-09-28 22:50:29,641 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 22:50:29,641 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 22:50:29,642 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.121.148:37283/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 22:50:29,905 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 22:50:29,905 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 22:50:29,905 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 22:50:29,905 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.121.148:37283/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 22:50:30,231 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 22:50:30.775545 16870 mini_ranger_kms.cc:310] Created kms service
I20260928 22:50:31.046406 16870 mini_ranger_kms.cc:326] Created kudu user
I20260928 22:50:31.098228 16870 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 22:50:31 dist-test-slave-56vw krb5kdc[27627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635827, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.16.121.148@KRBTEST.COM
I20260928 22:50:31.578907 16870 mini_ranger_kms.cc:384] Added ranger policy
I20260928 22:50:31.579037 16870 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ranger-kms:/tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 22:50:31.579212 16870 mini_ranger_kms.cc:240] Using host: 127.16.121.148
I20260928 22:50:31.583197 16870 mini_ranger_kms.cc:276] Ranger KMS PID: 28021
I20260928 22:50:31.583366 16870 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.16.121.148:36781
Sep 28, 2026 10:50:32 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 10:50:32 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 10:50:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 10:50:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 10:50:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 10:50:33 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 22:50:33.925480 16870 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.925s user 0.008s sys 0.149s
I20260928 22:50:33.925603 16870 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 22:50:33.925706 16870 mini_ranger_kms.cc:401] 127.16.121.148:36781/kms/v1/keys
Sep 28 22:50:34 dist-test-slave-56vw krb5kdc[27627](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790635834, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.121.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 22:50:34 dist-test-slave-56vw krb5kdc[27627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635834, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.121.148@KRBTEST.COM for HTTP/127.16.121.148@KRBTEST.COM
Sep 28 22:50:34 dist-test-slave-56vw krb5kdc[27627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635834, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.121.148@KRBTEST.COM for HTTP/127.16.121.148@KRBTEST.COM
Sep 28 22:50:37 dist-test-slave-56vw krb5kdc[27627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.16.121.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
I20260928 22:50:37.387831 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:35057
--webserver_interface=127.16.121.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:35969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:35057
--ranger_config_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.121.148:36781/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:50:37.486954 28071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:50:37.487211 28071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:50:37.487236 28071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:50:37.488662 28071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:50:37.488708 28071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:50:37.488721 28071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:50:37.488734 28071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:50:37.488746 28071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:50:37.490394 28071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:35969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/master-0/wal
--ranger_kms_url=127.16.121.148:36781/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:35057
--ranger_config_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:35057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.121.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:50:37.490655 28071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:50:37.490912 28071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:50:37.497120 28079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:50:37.497112 28077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:50:37.497115 28076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:50:37.497329 28071 server_base.cc:1034] running on GCE node
I20260928 22:50:37.497716 28071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:50:37.498090 28071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:50:37.499368 28071 hybrid_clock.cc:648] HybridClock initialized: now 1790635837499344 us; error 40 us; skew 500 ppm
Sep 28 22:50:37 dist-test-slave-56vw krb5kdc[27627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635837, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:50:37.502235 28071 init.cc:377] Logged in from keytab as kudu/127.16.121.190@KRBTEST.COM (short username kudu)
I20260928 22:50:37.502858 28071 webserver.cc:460] Webserver started at http://127.16.121.190:41411/ using document root <none> and password file <none>
I20260928 22:50:37.503139 28071 fs_manager.cc:359] Metadata directory not provided
I20260928 22:50:37.503193 28071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:50:37.503327 28071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 22:50:37 dist-test-slave-56vw krb5kdc[27627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635837, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for HTTP/127.16.121.148@KRBTEST.COM
I20260928 22:50:37.560336 28071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "febfcf520ba34508bbbb166bed86d07e"
format_stamp: "Formatted at 2026-09-28 22:50:37 on dist-test-slave-56vw"
server_key: "fec060ed4294687de53c769ee6ebf6ce"
server_key_iv: "1812c14b8ec186233d509e78d695e91e"
server_key_version: "kuduclusterkey@0"
I20260928 22:50:37.560808 28071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "febfcf520ba34508bbbb166bed86d07e"
format_stamp: "Formatted at 2026-09-28 22:50:37 on dist-test-slave-56vw"
server_key: "fec060ed4294687de53c769ee6ebf6ce"
server_key_iv: "1812c14b8ec186233d509e78d695e91e"
server_key_version: "kuduclusterkey@0"
I20260928 22:50:37.562644 28071 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 22:50:37.575095 28086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:50:37.575335 28071 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:50:37.575481 28071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "febfcf520ba34508bbbb166bed86d07e"
format_stamp: "Formatted at 2026-09-28 22:50:37 on dist-test-slave-56vw"
server_key: "fec060ed4294687de53c769ee6ebf6ce"
server_key_iv: "1812c14b8ec186233d509e78d695e91e"
server_key_version: "kuduclusterkey@0"
I20260928 22:50:37.575605 28071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:50:37.588788 28071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:50:37.590951 28071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:50:37.591179 28071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:50:37.596804 28071 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.190:35057
I20260928 22:50:37.596841 28138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.190:35057 every 8 connection(s)
I20260928 22:50:37.597245 28071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:50:37.598029 28139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:50:37.600632 28139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P febfcf520ba34508bbbb166bed86d07e: Bootstrap starting.
I20260928 22:50:37.601320 28139 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P febfcf520ba34508bbbb166bed86d07e: Neither blocks nor log segments found. Creating new log.
I20260928 22:50:37.601655 28139 log.cc:824] T 00000000000000000000000000000000 P febfcf520ba34508bbbb166bed86d07e: Log is configured to *not* fsync() on all Append() calls
I20260928 22:50:37.602401 28139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P febfcf520ba34508bbbb166bed86d07e: No bootstrap required, opened a new log
I20260928 22:50:37.604779 28139 raft_consensus.cc:348] T 00000000000000000000000000000000 P febfcf520ba34508bbbb166bed86d07e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "febfcf520ba34508bbbb166bed86d07e" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 35057 } }
I20260928 22:50:37.604981 28139 raft_consensus.cc:374] T 00000000000000000000000000000000 P febfcf520ba34508bbbb166bed86d07e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:50:37.605011 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 28071
I20260928 22:50:37.605054 28139 raft_consensus.cc:729] T 00000000000000000000000000000000 P febfcf520ba34508bbbb166bed86d07e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: febfcf520ba34508bbbb166bed86d07e, State: Initialized, Role: FOLLOWER
I20260928 22:50:37.605119 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:50:37.605255 28139 consensus_queue.cc:260] T 00000000000000000000000000000000 P febfcf520ba34508bbbb166bed86d07e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "febfcf520ba34508bbbb166bed86d07e" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 35057 } }
I20260928 22:50:37.605299 16870 external_mini_cluster.cc:1428] Setting key d4ea4ac768be4257cf165cb4ccc1dce4
I20260928 22:50:37.605356 28139 raft_consensus.cc:388] T 00000000000000000000000000000000 P febfcf520ba34508bbbb166bed86d07e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:50:37.605419 28139 raft_consensus.cc:482] T 00000000000000000000000000000000 P febfcf520ba34508bbbb166bed86d07e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:50:37.605482 28139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P febfcf520ba34508bbbb166bed86d07e [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:50:37.606315 28139 raft_consensus.cc:504] T 00000000000000000000000000000000 P febfcf520ba34508bbbb166bed86d07e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "febfcf520ba34508bbbb166bed86d07e" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 35057 } }
I20260928 22:50:37.606520 28139 leader_election.cc:302] T 00000000000000000000000000000000 P febfcf520ba34508bbbb166bed86d07e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: febfcf520ba34508bbbb166bed86d07e; no voters:
I20260928 22:50:37.606757 28139 leader_election.cc:288] T 00000000000000000000000000000000 P febfcf520ba34508bbbb166bed86d07e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:50:37.606813 28144 raft_consensus.cc:2781] T 00000000000000000000000000000000 P febfcf520ba34508bbbb166bed86d07e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:50:37.607038 28144 raft_consensus.cc:686] T 00000000000000000000000000000000 P febfcf520ba34508bbbb166bed86d07e [term 1 LEADER]: Becoming Leader. State: Replica: febfcf520ba34508bbbb166bed86d07e, State: Running, Role: LEADER
I20260928 22:50:37.607180 28139 sys_catalog.cc:564] T 00000000000000000000000000000000 P febfcf520ba34508bbbb166bed86d07e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:50:37.607792 28144 consensus_queue.cc:237] T 00000000000000000000000000000000 P febfcf520ba34508bbbb166bed86d07e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "febfcf520ba34508bbbb166bed86d07e" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 35057 } }
I20260928 22:50:37.608230 28145 sys_catalog.cc:455] T 00000000000000000000000000000000 P febfcf520ba34508bbbb166bed86d07e [sys.catalog]: SysCatalogTable state changed. Reason: New leader febfcf520ba34508bbbb166bed86d07e. Latest consensus state: current_term: 1 leader_uuid: "febfcf520ba34508bbbb166bed86d07e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "febfcf520ba34508bbbb166bed86d07e" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 35057 } } }
I20260928 22:50:37.608299 28145 sys_catalog.cc:458] T 00000000000000000000000000000000 P febfcf520ba34508bbbb166bed86d07e [sys.catalog]: This master's current role is: LEADER
I20260928 22:50:37.608686 28146 sys_catalog.cc:455] T 00000000000000000000000000000000 P febfcf520ba34508bbbb166bed86d07e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "febfcf520ba34508bbbb166bed86d07e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "febfcf520ba34508bbbb166bed86d07e" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 35057 } } }
I20260928 22:50:37.608736 28146 sys_catalog.cc:458] T 00000000000000000000000000000000 P febfcf520ba34508bbbb166bed86d07e [sys.catalog]: This master's current role is: LEADER
I20260928 22:50:37.608773 28139 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 22:50:37.608979 28156 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:50:37.612564 28156 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 22:50:37 dist-test-slave-56vw krb5kdc[27627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:50:37.619639 28156 catalog_manager.cc:1347] Generated new cluster ID: 25d86719555148538a33b51905e192e6
I20260928 22:50:37.619693 28156 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:50:37.625258 28143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:50:37.605809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51290 (local address 127.16.121.190:35057)
0928 22:50:37.606016 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 22:50:37.606026 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:50:37.606064 (+ 38us) server_negotiation.cc:407] Connection header received
0928 22:50:37.606197 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:50:37.606205 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:50:37.606270 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:50:37.606451 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:50:37.607240 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:50:37.608957 (+ 1717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:50:37.610739 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:50:37.613027 (+ 2288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:50:37.613230 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:50:37.617618 (+ 4388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:50:37.617623 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:50:37.617623 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:50:37.617638 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:50:37.619796 (+ 2158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:50:37.623550 (+ 3754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:50:37.623555 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:50:37.623555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:50:37.623714 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:50:37.623960 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:50:37.623960 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:50:37.623961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:50:37.624398 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:50:37.624734 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0928 22:50:37.624835 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 22:50:37.624993 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":91,"threads_started":1}
I20260928 22:50:37.632999 28156 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:50:37.633798 28156 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:50:37.638262 28156 catalog_manager.cc:5868] T 00000000000000000000000000000000 P febfcf520ba34508bbbb166bed86d07e: Generated new TSK 0
I20260928 22:50:37.638523 28156 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.121.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab.
Entry for principal kudu/127.16.121.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab.
I20260928 22:50:38.391448 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.121.148:36781/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.129:0
--local_ip_for_outbound_sockets=127.16.121.129
--webserver_interface=127.16.121.129
--webserver_port=0
--tserver_master_addrs=127.16.121.190:35057
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab
--principal=kudu/127.16.121.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:35969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:50:38.494691 28184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:50:38.494997 28184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:50:38.495034 28184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:50:38.496873 28184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:50:38.496937 28184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:50:38.496986 28184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.129
I20260928 22:50:38.498795 28184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:35969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.16.121.148:36781/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab
--principal=kudu/127.16.121.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.121.129
--webserver_port=0
--tserver_master_addrs=127.16.121.190:35057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.129
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:50:38.499126 28184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:50:38.499387 28184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:50:38.505975 28189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:50:38.506150 28192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:50:38.506215 28184 server_base.cc:1034] running on GCE node
W20260928 22:50:38.505977 28190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:50:38.506646 28184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:50:38.506994 28184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:50:38.508195 28184 hybrid_clock.cc:648] HybridClock initialized: now 1790635838508180 us; error 29 us; skew 500 ppm
Sep 28 22:50:38 dist-test-slave-56vw krb5kdc[27627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635838, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:50:38.511726 28184 init.cc:377] Logged in from keytab as kudu/127.16.121.129@KRBTEST.COM (short username kudu)
I20260928 22:50:38.512255 28184 webserver.cc:460] Webserver started at http://127.16.121.129:45263/ using document root <none> and password file <none>
I20260928 22:50:38.512503 28184 fs_manager.cc:359] Metadata directory not provided
I20260928 22:50:38.512575 28184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:50:38.512732 28184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 22:50:38 dist-test-slave-56vw krb5kdc[27627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635838, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for HTTP/127.16.121.148@KRBTEST.COM
I20260928 22:50:38.527575 28184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-0/data/instance:
uuid: "bcf9d9acbd244fe9974292440e82b50e"
format_stamp: "Formatted at 2026-09-28 22:50:38 on dist-test-slave-56vw"
server_key: "27e3d3797bf05c230ed9cf8e1ca8169f"
server_key_iv: "8527cd5f64d944d5534956fcc533f8f0"
server_key_version: "kuduclusterkey@0"
I20260928 22:50:38.528164 28184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance:
uuid: "bcf9d9acbd244fe9974292440e82b50e"
format_stamp: "Formatted at 2026-09-28 22:50:38 on dist-test-slave-56vw"
server_key: "27e3d3797bf05c230ed9cf8e1ca8169f"
server_key_iv: "8527cd5f64d944d5534956fcc533f8f0"
server_key_version: "kuduclusterkey@0"
I20260928 22:50:38.530063 28184 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 22:50:38.544206 28199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:50:38.544548 28184 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:50:38.544667 28184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-0/wal
uuid: "bcf9d9acbd244fe9974292440e82b50e"
format_stamp: "Formatted at 2026-09-28 22:50:38 on dist-test-slave-56vw"
server_key: "27e3d3797bf05c230ed9cf8e1ca8169f"
server_key_iv: "8527cd5f64d944d5534956fcc533f8f0"
server_key_version: "kuduclusterkey@0"
I20260928 22:50:38.544771 28184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:50:38.573114 28184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:50:38.575314 28184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:50:38.575492 28184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:50:38.575779 28184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:50:38.576220 28184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:50:38.576280 28184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:50:38.576351 28184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:50:38.576397 28184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:50:38.583810 28184 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.129:33873
I20260928 22:50:38.583977 28314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.129:33873 every 8 connection(s)
I20260928 22:50:38.584297 28184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
I20260928 22:50:38.589370 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 28184
I20260928 22:50:38.589493 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance
I20260928 22:50:38.589710 16870 external_mini_cluster.cc:1428] Setting key 0dc9f95351da760924f3e5a436823cb5
Sep 28 22:50:38 dist-test-slave-56vw krb5kdc[27627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635838, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:50:38.596129 28317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:50:38.585268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:35057 (local address 127.16.121.129:45913)
0928 22:50:38.585464 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 22:50:38.585520 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 22:50:38.586037 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:50:38.586419 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:50:38.586424 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:50:38.586706 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:50:38.586998 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:50:38.587006 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:50:38.587808 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:50:38.587810 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:50:38.589078 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:50:38.589081 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:50:38.589651 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:50:38.589653 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:50:38.589803 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:50:38.590562 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:50:38.590573 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:50:38.592194 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:50:38.594287 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:50:38.594295 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:50:38.594298 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:50:38.594615 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:50:38.594842 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:50:38.594843 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:50:38.594843 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:50:38.595003 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:50:38.595404 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:50:38.595412 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:50:38.595583 (+ 171us) client_negotiation.cc:769] Sending connection context
0928 22:50:38.595696 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 22:50:38.595864 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":92,"threads_started":1}
I20260928 22:50:38.596778 28318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:50:38.585289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:45913 (local address 127.16.121.190:35057)
0928 22:50:38.586182 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0928 22:50:38.586184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:50:38.586191 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:50:38.586213 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:50:38.586214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:50:38.586223 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:50:38.586277 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:50:38.587194 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:50:38.587707 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:50:38.589183 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:50:38.589505 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:50:38.589581 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:50:38.592293 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:50:38.592295 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:50:38.592295 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:50:38.592303 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:50:38.594184 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:50:38.594705 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:50:38.594706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:50:38.594706 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:50:38.594771 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:50:38.595159 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:50:38.595160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:50:38.595161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:50:38.595312 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:50:38.595376 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 22:50:38.596532 (+ 1156us) server_negotiation.cc:299] Negotiation successful
0928 22:50:38.596687 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":868,"thread_start_us":110,"threads_started":1}
I20260928 22:50:38.597708 28315 heartbeater.cc:347] Connected to a master server at 127.16.121.190:35057
I20260928 22:50:38.597862 28315 heartbeater.cc:464] Registering TS with master...
I20260928 22:50:38.598110 28315 heartbeater.cc:511] Master 127.16.121.190:35057 requested a full tablet report, sending...
I20260928 22:50:38.598706 28103 ts_manager.cc:194] Registered new tserver with Master: bcf9d9acbd244fe9974292440e82b50e (127.16.121.129:33873)
I20260928 22:50:38.599555 28103 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.129@KRBTEST.COM'} at 127.16.121.129:45913
WARNING: no policy specified for kudu/127.16.121.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab.
Entry for principal kudu/127.16.121.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab.
I20260928 22:50:38.625985 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.121.148:36781/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.130:0
--local_ip_for_outbound_sockets=127.16.121.130
--webserver_interface=127.16.121.130
--webserver_port=0
--tserver_master_addrs=127.16.121.190:35057
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab
--principal=kudu/127.16.121.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:35969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 22:50:38 dist-test-slave-56vw krb5kdc[27627](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790635838, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 22:50:38.753964 28321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:50:38.754372 28321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:50:38.754427 28321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:50:38.756364 28321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:50:38.756502 28321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:50:38.756603 28321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.130
I20260928 22:50:38.759141 28321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:35969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.16.121.148:36781/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab
--principal=kudu/127.16.121.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.121.130
--webserver_port=0
--tserver_master_addrs=127.16.121.190:35057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.130
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:50:38.759641 28321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:50:38.759982 28321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:50:38.766870 28329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:50:38.767228 28328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:50:38.766860 28331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:50:38.768606 28321 server_base.cc:1034] running on GCE node
I20260928 22:50:38.768829 28321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:50:38.769106 28321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:50:38.770268 28321 hybrid_clock.cc:648] HybridClock initialized: now 1790635838770242 us; error 40 us; skew 500 ppm
Sep 28 22:50:38 dist-test-slave-56vw krb5kdc[27627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635838, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:50:38.773787 28321 init.cc:377] Logged in from keytab as kudu/127.16.121.130@KRBTEST.COM (short username kudu)
I20260928 22:50:38.774395 28321 webserver.cc:460] Webserver started at http://127.16.121.130:38627/ using document root <none> and password file <none>
I20260928 22:50:38.774680 28321 fs_manager.cc:359] Metadata directory not provided
I20260928 22:50:38.774765 28321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:50:38.774971 28321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 22:50:38 dist-test-slave-56vw krb5kdc[27627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635838, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.130@KRBTEST.COM for HTTP/127.16.121.148@KRBTEST.COM
I20260928 22:50:38.788985 28321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-1/data/instance:
uuid: "33860e82437e4e01af17b2643788aa2b"
format_stamp: "Formatted at 2026-09-28 22:50:38 on dist-test-slave-56vw"
server_key: "e4ce982e4ba2760945d748f7d3b72f99"
server_key_iv: "0547e668035eb653bf9912204c625ac3"
server_key_version: "kuduclusterkey@0"
I20260928 22:50:38.789450 28321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance:
uuid: "33860e82437e4e01af17b2643788aa2b"
format_stamp: "Formatted at 2026-09-28 22:50:38 on dist-test-slave-56vw"
server_key: "e4ce982e4ba2760945d748f7d3b72f99"
server_key_iv: "0547e668035eb653bf9912204c625ac3"
server_key_version: "kuduclusterkey@0"
I20260928 22:50:38.791409 28321 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260928 22:50:38.804668 28338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:50:38.804929 28321 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:50:38.805018 28321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-1/wal
uuid: "33860e82437e4e01af17b2643788aa2b"
format_stamp: "Formatted at 2026-09-28 22:50:38 on dist-test-slave-56vw"
server_key: "e4ce982e4ba2760945d748f7d3b72f99"
server_key_iv: "0547e668035eb653bf9912204c625ac3"
server_key_version: "kuduclusterkey@0"
I20260928 22:50:38.805111 28321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:50:38.817238 28321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:50:38.819591 28321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:50:38.819826 28321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:50:38.820138 28321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:50:38.820560 28321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:50:38.820659 28321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:50:38.820730 28321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:50:38.820787 28321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:50:38.828549 28321 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.130:42009
I20260928 22:50:38.828969 28321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
I20260928 22:50:38.829154 28452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.130:42009 every 8 connection(s)
I20260928 22:50:38.831283 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 28321
I20260928 22:50:38.831413 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance
I20260928 22:50:38.831646 16870 external_mini_cluster.cc:1428] Setting key cee4b20461885c236ffd62ddf99d05b3
Sep 28 22:50:38 dist-test-slave-56vw krb5kdc[27627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635838, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.130@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:50:38.840240 28318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:50:38.829956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:36321 (local address 127.16.121.190:35057)
0928 22:50:38.830030 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 22:50:38.830032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:50:38.830205 (+ 173us) server_negotiation.cc:407] Connection header received
0928 22:50:38.830706 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:50:38.830707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:50:38.830722 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:50:38.830810 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:50:38.831482 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:50:38.832173 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:50:38.833428 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:50:38.833653 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:50:38.833732 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:50:38.836316 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:50:38.836318 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:50:38.836318 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:50:38.836328 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:50:38.838282 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:50:38.838867 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:50:38.838868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:50:38.838869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:50:38.838931 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:50:38.839304 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:50:38.839305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:50:38.839305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:50:38.839472 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:50:38.839522 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 22:50:38.839966 (+ 444us) server_negotiation.cc:299] Negotiation successful
0928 22:50:38.840077 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 22:50:38.840968 28455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:50:38.829886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:35057 (local address 127.16.121.130:36321)
0928 22:50:38.830066 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 22:50:38.830111 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 22:50:38.830595 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:50:38.830868 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:50:38.830871 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:50:38.831119 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:50:38.831362 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:50:38.831369 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:50:38.832464 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:50:38.832465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:50:38.833322 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:50:38.833324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:50:38.833753 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:50:38.833754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:50:38.833880 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:50:38.834553 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:50:38.834564 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:50:38.836206 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:50:38.838397 (+ 2191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:50:38.838408 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:50:38.838412 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:50:38.838770 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:50:38.839012 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:50:38.839013 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:50:38.839013 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:50:38.839215 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:50:38.839575 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:50:38.839581 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:50:38.839786 (+ 205us) client_negotiation.cc:769] Sending connection context
0928 22:50:38.840510 (+ 724us) client_negotiation.cc:240] Negotiation successful
0928 22:50:38.840712 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":80,"threads_started":1}
I20260928 22:50:38.841595 28453 heartbeater.cc:347] Connected to a master server at 127.16.121.190:35057
I20260928 22:50:38.841722 28453 heartbeater.cc:464] Registering TS with master...
I20260928 22:50:38.841926 28453 heartbeater.cc:511] Master 127.16.121.190:35057 requested a full tablet report, sending...
I20260928 22:50:38.842345 28103 ts_manager.cc:194] Registered new tserver with Master: 33860e82437e4e01af17b2643788aa2b (127.16.121.130:42009)
I20260928 22:50:38.842747 28103 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.130@KRBTEST.COM'} at 127.16.121.130:36321
WARNING: no policy specified for kudu/127.16.121.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab.
Entry for principal kudu/127.16.121.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab.
I20260928 22:50:38.866717 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.121.148:36781/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.131:0
--local_ip_for_outbound_sockets=127.16.121.131
--webserver_interface=127.16.121.131
--webserver_port=0
--tserver_master_addrs=127.16.121.190:35057
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab
--principal=kudu/127.16.121.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:35969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:50:38.994916 28458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:50:38.995286 28458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:50:38.995327 28458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:50:38.997457 28458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:50:38.999256 28458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:50:38.999316 28458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.131
I20260928 22:50:39.001924 28458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:35969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.16.121.148:36781/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab
--principal=kudu/127.16.121.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.121.131
--webserver_port=0
--tserver_master_addrs=127.16.121.190:35057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.131
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:50:39.002238 28458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:50:39.002635 28458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:50:39.008858 28463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:50:39.008919 28466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:50:39.008858 28464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:50:39.009550 28458 server_base.cc:1034] running on GCE node
I20260928 22:50:39.009833 28458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:50:39.010123 28458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:50:39.011294 28458 hybrid_clock.cc:648] HybridClock initialized: now 1790635839011290 us; error 47 us; skew 500 ppm
Sep 28 22:50:39 dist-test-slave-56vw krb5kdc[27627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635839, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:50:39.014328 28458 init.cc:377] Logged in from keytab as kudu/127.16.121.131@KRBTEST.COM (short username kudu)
I20260928 22:50:39.014906 28458 webserver.cc:460] Webserver started at http://127.16.121.131:42109/ using document root <none> and password file <none>
I20260928 22:50:39.015226 28458 fs_manager.cc:359] Metadata directory not provided
I20260928 22:50:39.015302 28458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:50:39.015470 28458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 22:50:39 dist-test-slave-56vw krb5kdc[27627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635839, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.131@KRBTEST.COM for HTTP/127.16.121.148@KRBTEST.COM
I20260928 22:50:39.029559 28458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-2/data/instance:
uuid: "374932baefd14ac882855c1f2fe24854"
format_stamp: "Formatted at 2026-09-28 22:50:39 on dist-test-slave-56vw"
server_key: "99ed109e65f550b7383fefa37fe16598"
server_key_iv: "36340e25d601382fabad0cd18824220d"
server_key_version: "kuduclusterkey@0"
I20260928 22:50:39.043828 28458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance:
uuid: "374932baefd14ac882855c1f2fe24854"
format_stamp: "Formatted at 2026-09-28 22:50:39 on dist-test-slave-56vw"
server_key: "99ed109e65f550b7383fefa37fe16598"
server_key_iv: "36340e25d601382fabad0cd18824220d"
server_key_version: "kuduclusterkey@0"
I20260928 22:50:39.055665 28458 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.002s sys 0.000s
I20260928 22:50:39.089864 28473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:50:39.090111 28458 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.000s sys 0.000s
I20260928 22:50:39.090211 28458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-2/wal
uuid: "374932baefd14ac882855c1f2fe24854"
format_stamp: "Formatted at 2026-09-28 22:50:39 on dist-test-slave-56vw"
server_key: "99ed109e65f550b7383fefa37fe16598"
server_key_iv: "36340e25d601382fabad0cd18824220d"
server_key_version: "kuduclusterkey@0"
I20260928 22:50:39.090327 28458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:50:39.109589 28458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:50:39.112370 28458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:50:39.112547 28458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:50:39.112794 28458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:50:39.113221 28458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:50:39.113276 28458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:50:39.113312 28458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:50:39.113335 28458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:50:39.121001 28458 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.131:41997
I20260928 22:50:39.121496 28458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
I20260928 22:50:39.124179 28586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.131:41997 every 8 connection(s)
I20260928 22:50:39.124449 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 28458
I20260928 22:50:39.124580 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance
I20260928 22:50:39.124797 16870 external_mini_cluster.cc:1428] Setting key b3c73ab44fdf7a9d1215c58955cb4fb2
Sep 28 22:50:39 dist-test-slave-56vw krb5kdc[27627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635839, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.131@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:50:39.136013 28318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:50:39.123911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:47485 (local address 127.16.121.190:35057)
0928 22:50:39.123960 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 22:50:39.123962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:50:39.124528 (+ 566us) server_negotiation.cc:407] Connection header received
0928 22:50:39.125604 (+ 1076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:50:39.125605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:50:39.125618 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:50:39.125649 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:50:39.126552 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:50:39.127352 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:50:39.128929 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:50:39.129224 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:50:39.129306 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:50:39.132092 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:50:39.132093 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:50:39.132094 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:50:39.132104 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:50:39.134092 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:50:39.134706 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:50:39.134707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:50:39.134707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:50:39.134778 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:50:39.135150 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:50:39.135151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:50:39.135152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:50:39.135303 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:50:39.135393 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 22:50:39.135776 (+ 383us) server_negotiation.cc:299] Negotiation successful
0928 22:50:39.135898 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 22:50:39.137602 28589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:50:39.123613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:35057 (local address 127.16.121.131:47485)
0928 22:50:39.124360 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0928 22:50:39.124404 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 22:50:39.125467 (+ 1063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:50:39.125853 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:50:39.125858 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:50:39.126127 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:50:39.126421 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:50:39.126428 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:50:39.127461 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:50:39.127462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:50:39.128810 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:50:39.128813 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:50:39.129336 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:50:39.129337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:50:39.129493 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:50:39.130220 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:50:39.130232 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:50:39.131961 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:50:39.134204 (+ 2243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:50:39.134216 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:50:39.134219 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:50:39.134603 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:50:39.134867 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:50:39.134868 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:50:39.134868 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:50:39.135032 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:50:39.135388 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:50:39.135396 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:50:39.135596 (+ 200us) client_negotiation.cc:769] Sending connection context
0928 22:50:39.137116 (+ 1520us) client_negotiation.cc:240] Negotiation successful
0928 22:50:39.137325 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":86,"threads_started":1}
I20260928 22:50:39.138296 28587 heartbeater.cc:347] Connected to a master server at 127.16.121.190:35057
I20260928 22:50:39.138441 28587 heartbeater.cc:464] Registering TS with master...
I20260928 22:50:39.138705 28587 heartbeater.cc:511] Master 127.16.121.190:35057 requested a full tablet report, sending...
I20260928 22:50:39.139132 28103 ts_manager.cc:194] Registered new tserver with Master: 374932baefd14ac882855c1f2fe24854 (127.16.121.131:41997)
I20260928 22:50:39.139667 28103 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.131@KRBTEST.COM'} at 127.16.121.131:47485
I20260928 22:50:39.148142 16870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:50:39.158347 28318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:50:39.150006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42278 (local address 127.16.121.190:35057)
0928 22:50:39.150059 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 22:50:39.150061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:50:39.150289 (+ 228us) server_negotiation.cc:407] Connection header received
0928 22:50:39.150346 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:50:39.150347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:50:39.150359 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:50:39.150417 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:50:39.151228 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:50:39.152025 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:50:39.153161 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:50:39.153458 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:50:39.153544 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:50:39.155220 (+ 1676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:50:39.155222 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:50:39.155222 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:50:39.155231 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:50:39.157024 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:50:39.157487 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:50:39.157488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:50:39.157488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:50:39.157548 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:50:39.157755 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:50:39.157756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:50:39.157756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:50:39.157889 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:50:39.157961 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 22:50:39.158166 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 22:50:39.158247 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 22:50:39.161629 28103 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42278:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:50:39.164072 28103 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:50:39.178732 28599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:50:39.166933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:33873 (local address 127.0.0.1:34494)
0928 22:50:39.172321 (+ 5388us) negotiation.cc:107] Waiting for socket to connect
0928 22:50:39.172331 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:50:39.172379 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:50:39.172665 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:50:39.172668 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:50:39.172670 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:50:39.172759 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:50:39.172761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:50:39.175362 (+ 2601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:50:39.175363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:50:39.176973 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:50:39.176976 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:50:39.178367 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:50:39.178368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:50:39.178442 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:50:39.178446 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:50:39.178607 (+ 161us) client_negotiation.cc:240] Negotiation successful
0928 22:50:39.178655 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5371,"mutex_wait_us":421,"thread_start_us":60,"threads_started":1}
I20260928 22:50:39.179261 28604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:50:39.167141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34494 (local address 127.16.121.129:33873)
0928 22:50:39.167284 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 22:50:39.167289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:50:39.172441 (+ 5152us) server_negotiation.cc:407] Connection header received
0928 22:50:39.172493 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:50:39.172496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:50:39.172511 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:50:39.172578 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:50:39.172830 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:50:39.173893 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:50:39.177070 (+ 3177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:50:39.177872 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:50:39.178182 (+ 310us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:50:39.178262 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 22:50:39.179112 (+ 850us) server_negotiation.cc:299] Negotiation successful
0928 22:50:39.179165 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":58,"threads_started":1}
I20260928 22:50:39.179571 28602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:50:39.166975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38800 (local address 127.16.121.130:42009)
0928 22:50:39.172038 (+ 5063us) server_negotiation.cc:206] Beginning negotiation
0928 22:50:39.172044 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 22:50:39.172056 (+ 12us) server_negotiation.cc:407] Connection header received
0928 22:50:39.172100 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:50:39.172102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:50:39.172121 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:50:39.172202 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:50:39.173997 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:50:39.175266 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:50:39.177096 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:50:39.177967 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:50:39.178084 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:50:39.178335 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0928 22:50:39.179437 (+ 1102us) server_negotiation.cc:299] Negotiation successful
0928 22:50:39.179495 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5016,"thread_start_us":102,"threads_started":1}
I20260928 22:50:39.178825 28600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:50:39.166740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.130:42009 (local address 127.0.0.1:38800)
0928 22:50:39.167683 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0928 22:50:39.167692 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:50:39.167727 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:50:39.173100 (+ 5373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:50:39.173102 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:50:39.173105 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:50:39.173159 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:50:39.173161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:50:39.175353 (+ 2192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:50:39.175355 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:50:39.176800 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:50:39.176803 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:50:39.178299 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:50:39.178300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:50:39.178389 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:50:39.178405 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 22:50:39.178478 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 22:50:39.178549 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":925,"mutex_wait_us":44,"thread_start_us":93,"threads_started":1}
I20260928 22:50:39.180418 28242 tablet_service.cc:1467] Processing CreateTablet for tablet 2826ee8f20cb4d99a9b3d9890904cd88 (DEFAULT_TABLE table=test-table [id=13c125047c984426b213b3071547e176]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:50:39.180442 28386 tablet_service.cc:1467] Processing CreateTablet for tablet 2826ee8f20cb4d99a9b3d9890904cd88 (DEFAULT_TABLE table=test-table [id=13c125047c984426b213b3071547e176]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:50:39.180807 28386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2826ee8f20cb4d99a9b3d9890904cd88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:50:39.180809 28242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2826ee8f20cb4d99a9b3d9890904cd88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:50:39.195258 28606 tablet_bootstrap.cc:492] T 2826ee8f20cb4d99a9b3d9890904cd88 P bcf9d9acbd244fe9974292440e82b50e: Bootstrap starting.
I20260928 22:50:39.196122 28606 tablet_bootstrap.cc:654] T 2826ee8f20cb4d99a9b3d9890904cd88 P bcf9d9acbd244fe9974292440e82b50e: Neither blocks nor log segments found. Creating new log.
I20260928 22:50:39.196475 28606 log.cc:824] T 2826ee8f20cb4d99a9b3d9890904cd88 P bcf9d9acbd244fe9974292440e82b50e: Log is configured to *not* fsync() on all Append() calls
I20260928 22:50:39.197616 28601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:50:39.166730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.131:41997 (local address 127.0.0.1:58992)
0928 22:50:39.167384 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0928 22:50:39.167426 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 22:50:39.167571 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:50:39.175925 (+ 8354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:50:39.175926 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:50:39.175928 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:50:39.175965 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:50:39.175966 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:50:39.194193 (+ 18227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:50:39.194195 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:50:39.196009 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:50:39.196013 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:50:39.197309 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:50:39.197311 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:50:39.197381 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:50:39.197386 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:50:39.197456 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 22:50:39.197516 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":69,"threads_started":1}
I20260928 22:50:39.198135 28603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:50:39.166979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58992 (local address 127.16.121.131:41997)
0928 22:50:39.175695 (+ 8716us) server_negotiation.cc:206] Beginning negotiation
0928 22:50:39.175700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:50:39.175711 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:50:39.175747 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:50:39.175749 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:50:39.175765 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:50:39.175844 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:50:39.180097 (+ 4253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:50:39.181186 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:50:39.196150 (+ 14964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:50:39.197107 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:50:39.197205 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:50:39.197268 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 22:50:39.198006 (+ 738us) server_negotiation.cc:299] Negotiation successful
0928 22:50:39.198053 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8673,"thread_start_us":110,"threads_started":1}
I20260928 22:50:39.199076 28507 tablet_service.cc:1467] Processing CreateTablet for tablet 2826ee8f20cb4d99a9b3d9890904cd88 (DEFAULT_TABLE table=test-table [id=13c125047c984426b213b3071547e176]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:50:39.199462 28507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2826ee8f20cb4d99a9b3d9890904cd88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:50:39.201390 28605 tablet_bootstrap.cc:492] T 2826ee8f20cb4d99a9b3d9890904cd88 P 33860e82437e4e01af17b2643788aa2b: Bootstrap starting.
I20260928 22:50:39.202533 28605 tablet_bootstrap.cc:654] T 2826ee8f20cb4d99a9b3d9890904cd88 P 33860e82437e4e01af17b2643788aa2b: Neither blocks nor log segments found. Creating new log.
I20260928 22:50:39.202826 28605 log.cc:824] T 2826ee8f20cb4d99a9b3d9890904cd88 P 33860e82437e4e01af17b2643788aa2b: Log is configured to *not* fsync() on all Append() calls
I20260928 22:50:39.203684 28606 tablet_bootstrap.cc:492] T 2826ee8f20cb4d99a9b3d9890904cd88 P bcf9d9acbd244fe9974292440e82b50e: No bootstrap required, opened a new log
I20260928 22:50:39.203768 28606 ts_tablet_manager.cc:1397] T 2826ee8f20cb4d99a9b3d9890904cd88 P bcf9d9acbd244fe9974292440e82b50e: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
Sep 28 22:50:39 dist-test-slave-56vw krb5kdc[27627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635838, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for HTTP/127.16.121.148@KRBTEST.COM
I20260928 22:50:39.206434 28606 raft_consensus.cc:348] T 2826ee8f20cb4d99a9b3d9890904cd88 P bcf9d9acbd244fe9974292440e82b50e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374932baefd14ac882855c1f2fe24854" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41997 } } peers { permanent_uuid: "33860e82437e4e01af17b2643788aa2b" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 42009 } } peers { permanent_uuid: "bcf9d9acbd244fe9974292440e82b50e" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33873 } }
I20260928 22:50:39.206621 28606 raft_consensus.cc:374] T 2826ee8f20cb4d99a9b3d9890904cd88 P bcf9d9acbd244fe9974292440e82b50e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:50:39.206655 28606 raft_consensus.cc:729] T 2826ee8f20cb4d99a9b3d9890904cd88 P bcf9d9acbd244fe9974292440e82b50e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf9d9acbd244fe9974292440e82b50e, State: Initialized, Role: FOLLOWER
I20260928 22:50:39.206805 28606 consensus_queue.cc:260] T 2826ee8f20cb4d99a9b3d9890904cd88 P bcf9d9acbd244fe9974292440e82b50e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374932baefd14ac882855c1f2fe24854" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41997 } } peers { permanent_uuid: "33860e82437e4e01af17b2643788aa2b" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 42009 } } peers { permanent_uuid: "bcf9d9acbd244fe9974292440e82b50e" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33873 } }
I20260928 22:50:39.207173 28606 ts_tablet_manager.cc:1428] T 2826ee8f20cb4d99a9b3d9890904cd88 P bcf9d9acbd244fe9974292440e82b50e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:50:39.207337 28315 heartbeater.cc:503] Master 127.16.121.190:35057 was elected leader, sending a full tablet report...
I20260928 22:50:39.211453 28609 tablet_bootstrap.cc:492] T 2826ee8f20cb4d99a9b3d9890904cd88 P 374932baefd14ac882855c1f2fe24854: Bootstrap starting.
I20260928 22:50:39.212539 28609 tablet_bootstrap.cc:654] T 2826ee8f20cb4d99a9b3d9890904cd88 P 374932baefd14ac882855c1f2fe24854: Neither blocks nor log segments found. Creating new log.
I20260928 22:50:39.212837 28609 log.cc:824] T 2826ee8f20cb4d99a9b3d9890904cd88 P 374932baefd14ac882855c1f2fe24854: Log is configured to *not* fsync() on all Append() calls
I20260928 22:50:39.213511 28605 tablet_bootstrap.cc:492] T 2826ee8f20cb4d99a9b3d9890904cd88 P 33860e82437e4e01af17b2643788aa2b: No bootstrap required, opened a new log
I20260928 22:50:39.213596 28605 ts_tablet_manager.cc:1397] T 2826ee8f20cb4d99a9b3d9890904cd88 P 33860e82437e4e01af17b2643788aa2b: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.001s
I20260928 22:50:39.220479 28605 raft_consensus.cc:348] T 2826ee8f20cb4d99a9b3d9890904cd88 P 33860e82437e4e01af17b2643788aa2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374932baefd14ac882855c1f2fe24854" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41997 } } peers { permanent_uuid: "33860e82437e4e01af17b2643788aa2b" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 42009 } } peers { permanent_uuid: "bcf9d9acbd244fe9974292440e82b50e" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33873 } }
I20260928 22:50:39.220832 28605 raft_consensus.cc:374] T 2826ee8f20cb4d99a9b3d9890904cd88 P 33860e82437e4e01af17b2643788aa2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:50:39.220932 28605 raft_consensus.cc:729] T 2826ee8f20cb4d99a9b3d9890904cd88 P 33860e82437e4e01af17b2643788aa2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33860e82437e4e01af17b2643788aa2b, State: Initialized, Role: FOLLOWER
I20260928 22:50:39.221133 28605 consensus_queue.cc:260] T 2826ee8f20cb4d99a9b3d9890904cd88 P 33860e82437e4e01af17b2643788aa2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374932baefd14ac882855c1f2fe24854" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41997 } } peers { permanent_uuid: "33860e82437e4e01af17b2643788aa2b" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 42009 } } peers { permanent_uuid: "bcf9d9acbd244fe9974292440e82b50e" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33873 } }
I20260928 22:50:39.221547 28605 ts_tablet_manager.cc:1428] T 2826ee8f20cb4d99a9b3d9890904cd88 P 33860e82437e4e01af17b2643788aa2b: Time spent starting tablet: real 0.008s user 0.002s sys 0.002s
I20260928 22:50:39.221752 28453 heartbeater.cc:503] Master 127.16.121.190:35057 was elected leader, sending a full tablet report...
I20260928 22:50:39.224655 28609 tablet_bootstrap.cc:492] T 2826ee8f20cb4d99a9b3d9890904cd88 P 374932baefd14ac882855c1f2fe24854: No bootstrap required, opened a new log
I20260928 22:50:39.224761 28609 ts_tablet_manager.cc:1397] T 2826ee8f20cb4d99a9b3d9890904cd88 P 374932baefd14ac882855c1f2fe24854: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20260928 22:50:39.232249 28609 raft_consensus.cc:348] T 2826ee8f20cb4d99a9b3d9890904cd88 P 374932baefd14ac882855c1f2fe24854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374932baefd14ac882855c1f2fe24854" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41997 } } peers { permanent_uuid: "33860e82437e4e01af17b2643788aa2b" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 42009 } } peers { permanent_uuid: "bcf9d9acbd244fe9974292440e82b50e" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33873 } }
I20260928 22:50:39.232517 28609 raft_consensus.cc:374] T 2826ee8f20cb4d99a9b3d9890904cd88 P 374932baefd14ac882855c1f2fe24854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:50:39.232553 28609 raft_consensus.cc:729] T 2826ee8f20cb4d99a9b3d9890904cd88 P 374932baefd14ac882855c1f2fe24854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 374932baefd14ac882855c1f2fe24854, State: Initialized, Role: FOLLOWER
I20260928 22:50:39.232736 28609 consensus_queue.cc:260] T 2826ee8f20cb4d99a9b3d9890904cd88 P 374932baefd14ac882855c1f2fe24854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374932baefd14ac882855c1f2fe24854" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41997 } } peers { permanent_uuid: "33860e82437e4e01af17b2643788aa2b" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 42009 } } peers { permanent_uuid: "bcf9d9acbd244fe9974292440e82b50e" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33873 } }
I20260928 22:50:39.233048 28609 ts_tablet_manager.cc:1428] T 2826ee8f20cb4d99a9b3d9890904cd88 P 374932baefd14ac882855c1f2fe24854: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20260928 22:50:39.233175 28587 heartbeater.cc:503] Master 127.16.121.190:35057 was elected leader, sending a full tablet report...
W20260928 22:50:39.329839 28454 tablet.cc:2369] T 2826ee8f20cb4d99a9b3d9890904cd88 P 33860e82437e4e01af17b2643788aa2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:50:39.335597 28316 tablet.cc:2369] T 2826ee8f20cb4d99a9b3d9890904cd88 P bcf9d9acbd244fe9974292440e82b50e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:50:39.374368 28588 tablet.cc:2369] T 2826ee8f20cb4d99a9b3d9890904cd88 P 374932baefd14ac882855c1f2fe24854: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 22:50:39 dist-test-slave-56vw krb5kdc[27627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635838, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for HTTP/127.16.121.148@KRBTEST.COM
I20260928 22:50:39.572225 28612 raft_consensus.cc:482] T 2826ee8f20cb4d99a9b3d9890904cd88 P 33860e82437e4e01af17b2643788aa2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:50:39.572388 28612 raft_consensus.cc:504] T 2826ee8f20cb4d99a9b3d9890904cd88 P 33860e82437e4e01af17b2643788aa2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374932baefd14ac882855c1f2fe24854" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41997 } } peers { permanent_uuid: "33860e82437e4e01af17b2643788aa2b" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 42009 } } peers { permanent_uuid: "bcf9d9acbd244fe9974292440e82b50e" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33873 } }
I20260928 22:50:39.572810 28612 leader_election.cc:288] T 2826ee8f20cb4d99a9b3d9890904cd88 P 33860e82437e4e01af17b2643788aa2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 374932baefd14ac882855c1f2fe24854 (127.16.121.131:41997), bcf9d9acbd244fe9974292440e82b50e (127.16.121.129:33873)
I20260928 22:50:39.577198 28603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:50:39.573172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:52351 (local address 127.16.121.131:41997)
0928 22:50:39.573235 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 22:50:39.573237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:50:39.573537 (+ 300us) server_negotiation.cc:407] Connection header received
0928 22:50:39.573652 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:50:39.573654 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:50:39.573675 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:50:39.573705 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:50:39.573931 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:50:39.574652 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:50:39.576229 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:50:39.576761 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:50:39.576816 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:50:39.576827 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 22:50:39.577075 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 22:50:39.577122 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 22:50:39.577692 28617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:50:39.572994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.131:41997 (local address 127.16.121.130:52351)
0928 22:50:39.573452 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0928 22:50:39.573465 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:50:39.573571 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:50:39.573776 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:50:39.573777 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:50:39.573778 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:50:39.573850 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:50:39.573853 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:50:39.574790 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:50:39.574790 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:50:39.576114 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:50:39.576117 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:50:39.576892 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:50:39.576893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:50:39.576967 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:50:39.576971 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:50:39.577553 (+ 582us) client_negotiation.cc:240] Negotiation successful
0928 22:50:39.577603 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":68,"threads_started":1}
I20260928 22:50:39.578213 28541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2826ee8f20cb4d99a9b3d9890904cd88" candidate_uuid: "33860e82437e4e01af17b2643788aa2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "374932baefd14ac882855c1f2fe24854" is_pre_election: true
I20260928 22:50:39.578388 28541 raft_consensus.cc:2454] T 2826ee8f20cb4d99a9b3d9890904cd88 P 374932baefd14ac882855c1f2fe24854 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33860e82437e4e01af17b2643788aa2b in term 0.
I20260928 22:50:39.578763 28339 leader_election.cc:302] T 2826ee8f20cb4d99a9b3d9890904cd88 P 33860e82437e4e01af17b2643788aa2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33860e82437e4e01af17b2643788aa2b, 374932baefd14ac882855c1f2fe24854; no voters:
I20260928 22:50:39.578914 28612 raft_consensus.cc:2781] T 2826ee8f20cb4d99a9b3d9890904cd88 P 33860e82437e4e01af17b2643788aa2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:50:39.578962 28612 raft_consensus.cc:482] T 2826ee8f20cb4d99a9b3d9890904cd88 P 33860e82437e4e01af17b2643788aa2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:50:39.578986 28612 raft_consensus.cc:3037] T 2826ee8f20cb4d99a9b3d9890904cd88 P 33860e82437e4e01af17b2643788aa2b [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:50:39.580029 28612 raft_consensus.cc:504] T 2826ee8f20cb4d99a9b3d9890904cd88 P 33860e82437e4e01af17b2643788aa2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374932baefd14ac882855c1f2fe24854" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41997 } } peers { permanent_uuid: "33860e82437e4e01af17b2643788aa2b" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 42009 } } peers { permanent_uuid: "bcf9d9acbd244fe9974292440e82b50e" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33873 } }
I20260928 22:50:39.580174 28612 leader_election.cc:288] T 2826ee8f20cb4d99a9b3d9890904cd88 P 33860e82437e4e01af17b2643788aa2b [CANDIDATE]: Term 1 election: Requested vote from peers 374932baefd14ac882855c1f2fe24854 (127.16.121.131:41997), bcf9d9acbd244fe9974292440e82b50e (127.16.121.129:33873)
I20260928 22:50:39.580565 28541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2826ee8f20cb4d99a9b3d9890904cd88" candidate_uuid: "33860e82437e4e01af17b2643788aa2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "374932baefd14ac882855c1f2fe24854"
I20260928 22:50:39.580648 28541 raft_consensus.cc:3037] T 2826ee8f20cb4d99a9b3d9890904cd88 P 374932baefd14ac882855c1f2fe24854 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:50:39.581518 28541 raft_consensus.cc:2454] T 2826ee8f20cb4d99a9b3d9890904cd88 P 374932baefd14ac882855c1f2fe24854 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33860e82437e4e01af17b2643788aa2b in term 1.
I20260928 22:50:39.581753 28339 leader_election.cc:302] T 2826ee8f20cb4d99a9b3d9890904cd88 P 33860e82437e4e01af17b2643788aa2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33860e82437e4e01af17b2643788aa2b, 374932baefd14ac882855c1f2fe24854; no voters:
I20260928 22:50:39.581873 28612 raft_consensus.cc:2781] T 2826ee8f20cb4d99a9b3d9890904cd88 P 33860e82437e4e01af17b2643788aa2b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:50:39.582038 28612 raft_consensus.cc:686] T 2826ee8f20cb4d99a9b3d9890904cd88 P 33860e82437e4e01af17b2643788aa2b [term 1 LEADER]: Becoming Leader. State: Replica: 33860e82437e4e01af17b2643788aa2b, State: Running, Role: LEADER
I20260928 22:50:39.582130 28612 consensus_queue.cc:237] T 2826ee8f20cb4d99a9b3d9890904cd88 P 33860e82437e4e01af17b2643788aa2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374932baefd14ac882855c1f2fe24854" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41997 } } peers { permanent_uuid: "33860e82437e4e01af17b2643788aa2b" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 42009 } } peers { permanent_uuid: "bcf9d9acbd244fe9974292440e82b50e" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33873 } }
I20260928 22:50:39.582803 28604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:50:39.573510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:49307 (local address 127.16.121.129:33873)
0928 22:50:39.573558 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 22:50:39.573560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:50:39.579208 (+ 5648us) server_negotiation.cc:407] Connection header received
0928 22:50:39.579323 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:50:39.579324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:50:39.579335 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:50:39.579367 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:50:39.579607 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:50:39.580348 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:50:39.581742 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:50:39.582364 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:50:39.582418 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:50:39.582428 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 22:50:39.582684 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 22:50:39.582732 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 22:50:39.583102 28616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:50:39.573307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:33873 (local address 127.16.121.130:49307)
0928 22:50:39.579123 (+ 5816us) negotiation.cc:107] Waiting for socket to connect
0928 22:50:39.579136 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:50:39.579241 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:50:39.579443 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:50:39.579443 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:50:39.579445 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:50:39.579514 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:50:39.579516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:50:39.580461 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:50:39.580461 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:50:39.581625 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:50:39.581628 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:50:39.582491 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:50:39.582492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:50:39.582566 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:50:39.582570 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:50:39.582921 (+ 351us) client_negotiation.cc:240] Negotiation successful
0928 22:50:39.582966 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5797,"thread_start_us":71,"threads_started":1}
I20260928 22:50:39.582966 28103 catalog_manager.cc:5495] T 2826ee8f20cb4d99a9b3d9890904cd88 P 33860e82437e4e01af17b2643788aa2b reported cstate change: term changed from 0 to 1, leader changed from <none> to 33860e82437e4e01af17b2643788aa2b (127.16.121.130). New cstate: current_term: 1 leader_uuid: "33860e82437e4e01af17b2643788aa2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374932baefd14ac882855c1f2fe24854" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33860e82437e4e01af17b2643788aa2b" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 42009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf9d9acbd244fe9974292440e82b50e" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33873 } health_report { overall_health: UNKNOWN } } }
I20260928 22:50:39.583436 28258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2826ee8f20cb4d99a9b3d9890904cd88" candidate_uuid: "33860e82437e4e01af17b2643788aa2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf9d9acbd244fe9974292440e82b50e" is_pre_election: true
I20260928 22:50:39.583604 28258 raft_consensus.cc:2454] T 2826ee8f20cb4d99a9b3d9890904cd88 P bcf9d9acbd244fe9974292440e82b50e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33860e82437e4e01af17b2643788aa2b in term 0.
I20260928 22:50:39.583513 28269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2826ee8f20cb4d99a9b3d9890904cd88" candidate_uuid: "33860e82437e4e01af17b2643788aa2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf9d9acbd244fe9974292440e82b50e"
I20260928 22:50:39.583663 28269 raft_consensus.cc:3037] T 2826ee8f20cb4d99a9b3d9890904cd88 P bcf9d9acbd244fe9974292440e82b50e [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:50:39.584612 28269 raft_consensus.cc:2454] T 2826ee8f20cb4d99a9b3d9890904cd88 P bcf9d9acbd244fe9974292440e82b50e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33860e82437e4e01af17b2643788aa2b in term 1.
I20260928 22:50:39.615257 28602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:50:39.611477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38810 (local address 127.16.121.130:42009)
0928 22:50:39.611623 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 22:50:39.611625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:50:39.611634 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:50:39.611678 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:50:39.611679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:50:39.611718 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:50:39.611792 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:50:39.611992 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:50:39.612751 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:50:39.613897 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:50:39.614228 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:50:39.614297 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:50:39.614538 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:50:39.614681 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:50:39.614840 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 22:50:39.615118 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 22:50:39.615145 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260928 22:50:39.618618 28269 raft_consensus.cc:1261] T 2826ee8f20cb4d99a9b3d9890904cd88 P bcf9d9acbd244fe9974292440e82b50e [term 1 FOLLOWER]: Refusing update from remote peer 33860e82437e4e01af17b2643788aa2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:50:39.619001 28618 consensus_queue.cc:1035] T 2826ee8f20cb4d99a9b3d9890904cd88 P 33860e82437e4e01af17b2643788aa2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf9d9acbd244fe9974292440e82b50e" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:50:39.621321 28541 raft_consensus.cc:1261] T 2826ee8f20cb4d99a9b3d9890904cd88 P 374932baefd14ac882855c1f2fe24854 [term 1 FOLLOWER]: Refusing update from remote peer 33860e82437e4e01af17b2643788aa2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:50:39.621733 28618 consensus_queue.cc:1035] T 2826ee8f20cb4d99a9b3d9890904cd88 P 33860e82437e4e01af17b2643788aa2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "374932baefd14ac882855c1f2fe24854" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:50:39.628839 28622 mvcc.cc:205] Tried to move back new op lower bound from 7334444399074332672 to 7334444398928797696. Current Snapshot: MvccSnapshot[applied={T|T < 7334444399074332672}]
I20260928 22:50:39.630882 28621 mvcc.cc:205] Tried to move back new op lower bound from 7334444399074332672 to 7334444398928797696. Current Snapshot: MvccSnapshot[applied={T|T < 7334444399074332672}]
I20260928 22:50:39.640547 28103 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:50:39.640679 28103 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:50:39.641549 28103 catalog_manager.cc:5782] T 00000000000000000000000000000000 P febfcf520ba34508bbbb166bed86d07e: Sending DeleteTablet for 3 replicas of tablet 2826ee8f20cb4d99a9b3d9890904cd88
I20260928 22:50:39.642091 28507 tablet_service.cc:1514] Processing DeleteTablet for tablet 2826ee8f20cb4d99a9b3d9890904cd88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:50:39 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:58992
I20260928 22:50:39.642280 28386 tablet_service.cc:1514] Processing DeleteTablet for tablet 2826ee8f20cb4d99a9b3d9890904cd88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:50:39 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:38800
I20260928 22:50:39.642472 28242 tablet_service.cc:1514] Processing DeleteTablet for tablet 2826ee8f20cb4d99a9b3d9890904cd88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:50:39 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:34494
I20260928 22:50:39.643370 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 28184
I20260928 22:50:39.648649 28630 tablet_replica.cc:330] T 2826ee8f20cb4d99a9b3d9890904cd88 P 374932baefd14ac882855c1f2fe24854: stopping tablet replica
I20260928 22:50:39.648836 28631 tablet_replica.cc:330] T 2826ee8f20cb4d99a9b3d9890904cd88 P 33860e82437e4e01af17b2643788aa2b: stopping tablet replica
W20260928 22:50:39.652921 28090 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.121.129:33873 (error 108)
I20260928 22:50:39.653388 28631 raft_consensus.cc:2229] T 2826ee8f20cb4d99a9b3d9890904cd88 P 33860e82437e4e01af17b2643788aa2b [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:50:39.653604 28630 raft_consensus.cc:2229] T 2826ee8f20cb4d99a9b3d9890904cd88 P 374932baefd14ac882855c1f2fe24854 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:50:39.653842 28631 raft_consensus.cc:2258] T 2826ee8f20cb4d99a9b3d9890904cd88 P 33860e82437e4e01af17b2643788aa2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:50:39.654006 28630 raft_consensus.cc:2258] T 2826ee8f20cb4d99a9b3d9890904cd88 P 374932baefd14ac882855c1f2fe24854 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:50:39.654767 28630 ts_tablet_manager.cc:1905] T 2826ee8f20cb4d99a9b3d9890904cd88 P 374932baefd14ac882855c1f2fe24854: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:50:39.655230 28631 ts_tablet_manager.cc:1905] T 2826ee8f20cb4d99a9b3d9890904cd88 P 33860e82437e4e01af17b2643788aa2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:50:39.658710 28630 ts_tablet_manager.cc:1918] T 2826ee8f20cb4d99a9b3d9890904cd88 P 374932baefd14ac882855c1f2fe24854: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:50:39.658810 28630 log.cc:1196] T 2826ee8f20cb4d99a9b3d9890904cd88 P 374932baefd14ac882855c1f2fe24854: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-2/wal/wals/2826ee8f20cb4d99a9b3d9890904cd88
I20260928 22:50:39.658922 28631 ts_tablet_manager.cc:1918] T 2826ee8f20cb4d99a9b3d9890904cd88 P 33860e82437e4e01af17b2643788aa2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:50:39.659024 28631 log.cc:1196] T 2826ee8f20cb4d99a9b3d9890904cd88 P 33860e82437e4e01af17b2643788aa2b: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790635733329745-16870-0/minicluster-data/ts-1/wal/wals/2826ee8f20cb4d99a9b3d9890904cd88
I20260928 22:50:39.659301 28630 ts_tablet_manager.cc:1939] T 2826ee8f20cb4d99a9b3d9890904cd88 P 374932baefd14ac882855c1f2fe24854: Deleting consensus metadata
I20260928 22:50:39.659798 28087 catalog_manager.cc:4841] TS 374932baefd14ac882855c1f2fe24854 (127.16.121.131:41997): tablet 2826ee8f20cb4d99a9b3d9890904cd88 (table test-table [id=13c125047c984426b213b3071547e176]) successfully deleted
I20260928 22:50:39.659943 28631 ts_tablet_manager.cc:1939] T 2826ee8f20cb4d99a9b3d9890904cd88 P 33860e82437e4e01af17b2643788aa2b: Deleting consensus metadata
I20260928 22:50:39.660336 28090 catalog_manager.cc:4841] TS 33860e82437e4e01af17b2643788aa2b (127.16.121.130:42009): tablet 2826ee8f20cb4d99a9b3d9890904cd88 (table test-table [id=13c125047c984426b213b3071547e176]) successfully deleted
W20260928 22:50:39.664268 28601 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:50:39.658030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:33873 (local address 127.0.0.1:34510)
0928 22:50:39.658084 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 22:50:39.658099 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 22:50:39.658144 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:50:39.664213 (+ 6069us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.129:33873: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 22:50:39.664532 28090 catalog_manager.cc:4584] TS bcf9d9acbd244fe9974292440e82b50e (127.16.121.129:33873): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2826ee8f20cb4d99a9b3d9890904cd88: Network error: Client connection negotiation failed: client connection to 127.16.121.129:33873: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 22:50:39.665066 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 28321
I20260928 22:50:39.671533 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 28458
I20260928 22:50:39.677522 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 28071
I20260928 22:50:39.683841 16870 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 22:50:40.028874 16870 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 22:50:40.028995 16870 mini_ranger.cc:66] Stopping Ranger...
I20260928 22:50:40.374032 16870 mini_ranger.cc:68] Stopped Ranger
2026-09-28 22:50:40.374 UTC [27657] LOG: received smart shutdown request
2026-09-28 22:50:40.378 UTC [27657] LOG: background worker "logical replication launcher" (PID 27666) exited with exit code 1
2026-09-28 22:50:40.378 UTC [27661] LOG: shutting down
2026-09-28 22:50:40.584 UTC [27657] LOG: database system is shut down
2026-09-28T22:50:40Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49817 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:50:40 dist-test-slave-56vw krb5kdc[28642](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:50:40 dist-test-slave-56vw krb5kdc[28642](info): set up 2 sockets
Sep 28 22:50:40 dist-test-slave-56vw krb5kdc[28642](info): commencing operation
krb5kdc: starting...
W20260928 22:50:41.795428 16870 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Sep 28 22:50:41 dist-test-slave-56vw krb5kdc[28642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:50:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:50:41Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.16.121.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.16.121.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.16.121.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/rangeradmin_127.16.121.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.16.121.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/rangeradmin_127.16.121.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.16.121.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.16.121.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.16.121.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/rangerlookup_127.16.121.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.16.121.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/rangerlookup_127.16.121.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.121.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.16.121.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.121.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/HTTP_127.16.121.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.121.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/HTTP_127.16.121.148@KRBTEST.COM.keytab.
I20260928 22:50:41.974572 16870 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskhFq4Cy/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/postgres -l logfile start
2026-09-28 22:50:45.152 UTC [28672] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 22:50:45.152 UTC [28672] LOG: listening on IPv4 address "127.16.121.148", port 40269
2026-09-28 22:50:45.158 UTC [28672] LOG: listening on Unix socket "/tmp/.s.PGSQL.40269"
2026-09-28 22:50:45.183 UTC [28675] LOG: database system was shut down at 2026-09-28 22:50:43 UTC
2026-09-28 22:50:45.187 UTC [28672] LOG: database system is ready to accept connections
I20260928 22:50:46.155452 16870 mini_postgres.cc:96] Postgres bound to 40269
2026-09-28 22:50:46.161 UTC [28683] FATAL: database "slave" does not exist
127.16.121.148:40269 - accepting connections
I20260928 22:50:46.162359 16870 mini_ranger.cc:161] Starting Ranger...
I20260928 22:50:46.178184 16870 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 22:50:46.847570 16870 mini_ranger.cc:87] Created ranger Postgres database
I20260928 22:50:46.847694 16870 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ranger-admin
2026-09-28 22:50:47,137 [I] DB FLAVOR :POSTGRES
2026-09-28 22:50:47,138 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 22:50:47,138 [I] Checking connection..
2026-09-28 22:50:47,138 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.121.148:40269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 22:50:47,398 [I] Checking connection passed.
2026-09-28 22:50:47,399 [I] --------- Verifying version history table ---------
2026-09-28 22:50:47,399 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.121.148:40269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 22:50:47,653 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 22:50:47,653 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.121.148:40269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 22:50:47,931 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 22:50:47,931 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 22:50:47,931 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.121.148:40269/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 22:50:48.192 UTC [28747] WARNING: there is no transaction in progress
2026-09-28 22:50:48,200 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 22:50:48,201 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.121.148:40269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 22:50:48,461 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 22:50:48,461 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 22:50:48,461 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.121.148:40269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 22:50:48,698 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.121.148:40269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 22:50:48,930 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.121.148:40269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-56vw.c.gcp-upstream.internal','N') ;"
2026-09-28 22:50:49,187 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.121.148:40269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 22:50:49,448 [I] Table x_portal_user does not exist in database ranger
2026-09-28 22:50:49,449 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.121.148:40269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 22:50:49,722 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 22:50:49,723 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.121.148:40269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 22:50:49,989 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.121.148:40269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 22:50:50,235 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 22:50:50,235 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.121.148:40269/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 22:50:51.293 UTC [28855] WARNING: there is no transaction in progress
2026-09-28 22:50:51.311 UTC [28855] WARNING: there is no transaction in progress
2026-09-28 22:50:51.330 UTC [28855] WARNING: there is no transaction in progress
2026-09-28 22:50:51.349 UTC [28855] WARNING: there is no transaction in progress
2026-09-28 22:50:51.370 UTC [28855] WARNING: there is no transaction in progress
2026-09-28 22:50:51.393 UTC [28855] WARNING: there is no transaction in progress
2026-09-28 22:50:51.458 UTC [28855] WARNING: there is no transaction in progress
2026-09-28 22:50:51.466 UTC [28855] WARNING: there is no transaction in progress
2026-09-28 22:50:51.483 UTC [28855] WARNING: there is no transaction in progress
2026-09-28 22:50:51.495 UTC [28855] WARNING: there is no transaction in progress
2026-09-28 22:50:51.507 UTC [28855] WARNING: there is no transaction in progress
2026-09-28 22:50:51.523 UTC [28855] WARNING: there is no transaction in progress
2026-09-28 22:50:51.535 UTC [28855] WARNING: there is no transaction in progress
2026-09-28 22:50:51.543 UTC [28855] WARNING: there is no transaction in progress
2026-09-28 22:50:51.552 UTC [28855] WARNING: there is no transaction in progress
2026-09-28 22:50:51.560 UTC [28855] WARNING: there is no transaction in progress
2026-09-28 22:50:52,343 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 22:50:52,343 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.121.148:40269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 22:50:52,606 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 22:50:52,607 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.121.148:40269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 22:50:52,879 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 22:50:52,879 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.121.148:40269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 22:50:53,127 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 22:50:53,127 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.121.148:40269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 22:50:53,387 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.121.148:40269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 22:50:53,637 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.121.148:40269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-56vw.c.gcp-upstream.internal';"
2026-09-28 22:50:53,882 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 22:50:53,883 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.121.148:40269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 22:50:54,128 [I] DB_PATCHES have already been applied
I20260928 22:50:54.135591 16870 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ranger-admin:/tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskhFq4Cy/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 22:50:54.135690 16870 mini_ranger.cc:193] Using host: 127.16.121.148
I20260928 22:50:54.139386 16870 mini_ranger.cc:227] Ranger admin URL: http://127.16.121.148:44239
Sep 28, 2026 10:50:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 10:50:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 10:50:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 10:50:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 10:50:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 10:50:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 10:50:55 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 22:50:55.361071 16870 mini_ranger.cc:160] Time spent starting Ranger: real 9.199s user 0.000s sys 0.010s
Sep 28, 2026 10:51:16 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 10:51:16 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 10:51:16 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 10:51:16 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 10:51:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 10:51:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 10:51:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 10:51:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 10:51:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 10:51:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 10:51:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 10:51:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 10:51:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 10:51:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 10:51:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 10:51:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 10:51:16 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 10:51:17 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 22:51:18.427822 16870 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.16.121.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.16.121.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.16.121.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/rangerkms_127.16.121.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.16.121.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/rangerkms_127.16.121.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.121.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.16.121.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.121.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/HTTP_127.16.121.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.121.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/HTTP_127.16.121.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 22:51:18 dist-test-slave-56vw krb5kdc[28642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635878, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 22:51:18.518067 16870 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 22:51:18.528121 16870 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 22:51:19.541059 16870 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 22:51:19.541147 16870 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ranger-kms
I20260928 22:51:19.541172 16870 mini_ranger_kms.cc:227] Using postgres at 127.16.121.148:40269
2026-09-28 22:51:19,720 [I] DB FLAVOR :POSTGRES
2026-09-28 22:51:19,721 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 22:51:19,721 [I] Checking connection
2026-09-28 22:51:19,721 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.121.148:40269/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 22:51:19,998 [I] connection success
2026-09-28 22:51:19,998 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 22:51:19,998 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 22:51:19,998 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.121.148:40269/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 22:51:20,244 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 22:51:20,244 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 22:51:20,244 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 22:51:20,244 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.121.148:40269/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 22:51:20,537 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 22:51:21.073395 16870 mini_ranger_kms.cc:310] Created kms service
I20260928 22:51:21.262035 16870 mini_ranger_kms.cc:326] Created kudu user
I20260928 22:51:21.309580 16870 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 22:51:21 dist-test-slave-56vw krb5kdc[28642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635878, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.16.121.148@KRBTEST.COM
I20260928 22:51:21.717644 16870 mini_ranger_kms.cc:384] Added ranger policy
I20260928 22:51:21.717762 16870 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ranger-kms:/tmp/dist-test-taskhFq4Cy/build/release/bin/postgresql.jar:/tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskhFq4Cy/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 22:51:21.717780 16870 mini_ranger_kms.cc:240] Using host: 127.16.121.148
I20260928 22:51:21.721503 16870 mini_ranger_kms.cc:276] Ranger KMS PID: 29036
I20260928 22:51:21.721597 16870 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.16.121.148:51789
Sep 28, 2026 10:51:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 10:51:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 10:51:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 10:51:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 10:51:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 10:51:23 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 22:51:24.018774 16870 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.501s user 0.000s sys 0.161s
I20260928 22:51:24.018886 16870 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 22:51:24.018935 16870 mini_ranger_kms.cc:401] 127.16.121.148:51789/kms/v1/keys
Sep 28 22:51:24 dist-test-slave-56vw krb5kdc[28642](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790635884, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.121.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 22:51:24 dist-test-slave-56vw krb5kdc[28642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635884, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.121.148@KRBTEST.COM for HTTP/127.16.121.148@KRBTEST.COM
Sep 28 22:51:24 dist-test-slave-56vw krb5kdc[28642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635884, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.121.148@KRBTEST.COM for HTTP/127.16.121.148@KRBTEST.COM
Sep 28 22:51:27 dist-test-slave-56vw krb5kdc[28642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.16.121.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
I20260928 22:51:27.363787 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:38111
--webserver_interface=127.16.121.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:45883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:38111
--ranger_config_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.121.148:51789/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:51:27.453418 29086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:51:27.453678 29086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:51:27.453715 29086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:51:27.454977 29086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:51:27.455049 29086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:51:27.455112 29086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:51:27.455139 29086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:51:27.455161 29086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:51:27.456907 29086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:45883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/master-0/wal
--ranger_kms_url=127.16.121.148:51789/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:38111
--ranger_config_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:38111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.121.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:51:27.457227 29086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:51:27.457456 29086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:51:27.463521 29091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:51:27.463557 29092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:51:27.463555 29094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:51:27.463881 29086 server_base.cc:1034] running on GCE node
I20260928 22:51:27.464172 29086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:51:27.464488 29086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:51:27.465693 29086 hybrid_clock.cc:648] HybridClock initialized: now 1790635887465655 us; error 54 us; skew 500 ppm
Sep 28 22:51:27 dist-test-slave-56vw krb5kdc[28642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635887, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:51:27.468479 29086 init.cc:377] Logged in from keytab as kudu/127.16.121.190@KRBTEST.COM (short username kudu)
I20260928 22:51:27.468915 29086 webserver.cc:460] Webserver started at http://127.16.121.190:39329/ using document root <none> and password file <none>
I20260928 22:51:27.469113 29086 fs_manager.cc:359] Metadata directory not provided
I20260928 22:51:27.469149 29086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:51:27.469241 29086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 22:51:27 dist-test-slave-56vw krb5kdc[28642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635887, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for HTTP/127.16.121.148@KRBTEST.COM
I20260928 22:51:27.528532 29086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "a50a70da076c47daa85d718844995547"
format_stamp: "Formatted at 2026-09-28 22:51:27 on dist-test-slave-56vw"
server_key: "0a81a4ad280c0d143e786aa79deb982b"
server_key_iv: "4d4a3c5aa1a68a1d99396937e679a2ec"
server_key_version: "kuduclusterkey@0"
I20260928 22:51:27.529048 29086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "a50a70da076c47daa85d718844995547"
format_stamp: "Formatted at 2026-09-28 22:51:27 on dist-test-slave-56vw"
server_key: "0a81a4ad280c0d143e786aa79deb982b"
server_key_iv: "4d4a3c5aa1a68a1d99396937e679a2ec"
server_key_version: "kuduclusterkey@0"
I20260928 22:51:27.531036 29086 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 22:51:27.547348 29101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:27.547614 29086 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:51:27.547724 29086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "a50a70da076c47daa85d718844995547"
format_stamp: "Formatted at 2026-09-28 22:51:27 on dist-test-slave-56vw"
server_key: "0a81a4ad280c0d143e786aa79deb982b"
server_key_iv: "4d4a3c5aa1a68a1d99396937e679a2ec"
server_key_version: "kuduclusterkey@0"
I20260928 22:51:27.547825 29086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:51:27.573076 29086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:51:27.575551 29086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:51:27.575717 29086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:51:27.580854 29086 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.190:38111
I20260928 22:51:27.580981 29153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.190:38111 every 8 connection(s)
I20260928 22:51:27.581333 29086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:51:27.582162 29154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:51:27.582290 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 29086
I20260928 22:51:27.582391 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:51:27.582614 16870 external_mini_cluster.cc:1428] Setting key 20ab8e870226273e1452408db7c1b201
I20260928 22:51:27.585201 29154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a50a70da076c47daa85d718844995547: Bootstrap starting.
I20260928 22:51:27.585819 29154 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a50a70da076c47daa85d718844995547: Neither blocks nor log segments found. Creating new log.
I20260928 22:51:27.586148 29154 log.cc:824] T 00000000000000000000000000000000 P a50a70da076c47daa85d718844995547: Log is configured to *not* fsync() on all Append() calls
I20260928 22:51:27.587244 29154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a50a70da076c47daa85d718844995547: No bootstrap required, opened a new log
I20260928 22:51:27.589525 29154 raft_consensus.cc:348] T 00000000000000000000000000000000 P a50a70da076c47daa85d718844995547 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a50a70da076c47daa85d718844995547" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 38111 } }
I20260928 22:51:27.589705 29154 raft_consensus.cc:374] T 00000000000000000000000000000000 P a50a70da076c47daa85d718844995547 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:51:27.589740 29154 raft_consensus.cc:729] T 00000000000000000000000000000000 P a50a70da076c47daa85d718844995547 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a50a70da076c47daa85d718844995547, State: Initialized, Role: FOLLOWER
I20260928 22:51:27.589911 29154 consensus_queue.cc:260] T 00000000000000000000000000000000 P a50a70da076c47daa85d718844995547 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a50a70da076c47daa85d718844995547" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 38111 } }
I20260928 22:51:27.590036 29154 raft_consensus.cc:388] T 00000000000000000000000000000000 P a50a70da076c47daa85d718844995547 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:51:27.590094 29154 raft_consensus.cc:482] T 00000000000000000000000000000000 P a50a70da076c47daa85d718844995547 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:51:27.590153 29154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a50a70da076c47daa85d718844995547 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:51:27.590732 29154 raft_consensus.cc:504] T 00000000000000000000000000000000 P a50a70da076c47daa85d718844995547 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a50a70da076c47daa85d718844995547" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 38111 } }
I20260928 22:51:27.590859 29154 leader_election.cc:302] T 00000000000000000000000000000000 P a50a70da076c47daa85d718844995547 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a50a70da076c47daa85d718844995547; no voters:
I20260928 22:51:27.591079 29154 leader_election.cc:288] T 00000000000000000000000000000000 P a50a70da076c47daa85d718844995547 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:51:27.591157 29159 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a50a70da076c47daa85d718844995547 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:51:27.591341 29159 raft_consensus.cc:686] T 00000000000000000000000000000000 P a50a70da076c47daa85d718844995547 [term 1 LEADER]: Becoming Leader. State: Replica: a50a70da076c47daa85d718844995547, State: Running, Role: LEADER
I20260928 22:51:27.591544 29159 consensus_queue.cc:237] T 00000000000000000000000000000000 P a50a70da076c47daa85d718844995547 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a50a70da076c47daa85d718844995547" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 38111 } }
I20260928 22:51:27.591430 29154 sys_catalog.cc:564] T 00000000000000000000000000000000 P a50a70da076c47daa85d718844995547 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:51:27.592435 29160 sys_catalog.cc:455] T 00000000000000000000000000000000 P a50a70da076c47daa85d718844995547 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a50a70da076c47daa85d718844995547" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a50a70da076c47daa85d718844995547" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 38111 } } }
I20260928 22:51:27.592514 29160 sys_catalog.cc:458] T 00000000000000000000000000000000 P a50a70da076c47daa85d718844995547 [sys.catalog]: This master's current role is: LEADER
I20260928 22:51:27.593132 29161 sys_catalog.cc:455] T 00000000000000000000000000000000 P a50a70da076c47daa85d718844995547 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a50a70da076c47daa85d718844995547. Latest consensus state: current_term: 1 leader_uuid: "a50a70da076c47daa85d718844995547" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a50a70da076c47daa85d718844995547" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 38111 } } }
I20260928 22:51:27.593263 29161 sys_catalog.cc:458] T 00000000000000000000000000000000 P a50a70da076c47daa85d718844995547 [sys.catalog]: This master's current role is: LEADER
I20260928 22:51:27.593335 29154 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 22:51:27.593420 29168 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 28 22:51:27 dist-test-slave-56vw krb5kdc[28642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:51:27.597630 29168 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:51:27.603672 29168 catalog_manager.cc:1347] Generated new cluster ID: 4ddeb9ca1c7f42bab3f55ce382a93677
I20260928 22:51:27.603729 29168 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:51:27.610845 29157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:27.583216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38254 (local address 127.16.121.190:38111)
0928 22:51:27.586464 (+ 3248us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:27.586472 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:27.586500 (+ 28us) server_negotiation.cc:407] Connection header received
0928 22:51:27.587208 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:27.587218 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:27.587289 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:27.587479 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:51:27.588287 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:27.589766 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:27.590994 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:27.591770 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:27.591880 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:27.596729 (+ 4849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:51:27.596781 (+ 52us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:51:27.596782 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:51:27.596886 (+ 104us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:51:27.600326 (+ 3440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:27.600835 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:27.600842 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:27.600843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:27.600990 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:27.601232 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:27.601233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:27.601233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:27.602622 (+ 1389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:51:27.610301 (+ 7679us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:27.610412 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 22:51:27.610602 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3194,"thread_start_us":82,"threads_started":1}
I20260928 22:51:27.655159 29168 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:51:27.655927 29168 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:51:27.662606 29168 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a50a70da076c47daa85d718844995547: Generated new TSK 0
I20260928 22:51:27.662814 29168 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.121.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab.
Entry for principal kudu/127.16.121.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab.
I20260928 22:51:28.384176 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.121.148:51789/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.129:0
--local_ip_for_outbound_sockets=127.16.121.129
--webserver_interface=127.16.121.129
--webserver_port=0
--tserver_master_addrs=127.16.121.190:38111
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab
--principal=kudu/127.16.121.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:45883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:51:28.503611 29199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:51:28.503876 29199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:51:28.503924 29199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:51:28.505376 29199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:51:28.505452 29199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:51:28.505507 29199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.129
I20260928 22:51:28.507725 29199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:45883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.16.121.148:51789/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab
--principal=kudu/127.16.121.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.121.129
--webserver_port=0
--tserver_master_addrs=127.16.121.190:38111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.129
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:51:28.508036 29199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:51:28.508348 29199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 22:51:28.515034 29199 server_base.cc:1034] running on GCE node
W20260928 22:51:28.515013 29205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:51:28.515013 29207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:51:28.515288 29204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:51:28.515640 29199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:51:28.515941 29199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:51:28.517159 29199 hybrid_clock.cc:648] HybridClock initialized: now 1790635888517127 us; error 46 us; skew 500 ppm
Sep 28 22:51:28 dist-test-slave-56vw krb5kdc[28642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635888, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:51:28.520402 29199 init.cc:377] Logged in from keytab as kudu/127.16.121.129@KRBTEST.COM (short username kudu)
I20260928 22:51:28.520984 29199 webserver.cc:460] Webserver started at http://127.16.121.129:41591/ using document root <none> and password file <none>
I20260928 22:51:28.521325 29199 fs_manager.cc:359] Metadata directory not provided
I20260928 22:51:28.521400 29199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:51:28.521538 29199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 22:51:28 dist-test-slave-56vw krb5kdc[28642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635888, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for HTTP/127.16.121.148@KRBTEST.COM
I20260928 22:51:28.539830 29199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-0/data/instance:
uuid: "627f99588e0341d88d2272de309ddd6d"
format_stamp: "Formatted at 2026-09-28 22:51:28 on dist-test-slave-56vw"
server_key: "5ad5568f0bdf58888af676083ab46d76"
server_key_iv: "bc8c98d5a9ec58fd4582612c410cddc0"
server_key_version: "kuduclusterkey@0"
I20260928 22:51:28.540370 29199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance:
uuid: "627f99588e0341d88d2272de309ddd6d"
format_stamp: "Formatted at 2026-09-28 22:51:28 on dist-test-slave-56vw"
server_key: "5ad5568f0bdf58888af676083ab46d76"
server_key_iv: "bc8c98d5a9ec58fd4582612c410cddc0"
server_key_version: "kuduclusterkey@0"
I20260928 22:51:28.542743 29199 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 22:51:28.557560 29216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:28.557894 29199 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:51:28.558028 29199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-0/wal
uuid: "627f99588e0341d88d2272de309ddd6d"
format_stamp: "Formatted at 2026-09-28 22:51:28 on dist-test-slave-56vw"
server_key: "5ad5568f0bdf58888af676083ab46d76"
server_key_iv: "bc8c98d5a9ec58fd4582612c410cddc0"
server_key_version: "kuduclusterkey@0"
I20260928 22:51:28.558136 29199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:51:28.577346 29199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:51:28.580129 29199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:51:28.580379 29199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:51:28.582451 29199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:51:28.582861 29199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:51:28.582926 29199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:28.583003 29199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:51:28.583061 29199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:28.588950 29199 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.129:39337
I20260928 22:51:28.589141 29329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.129:39337 every 8 connection(s)
I20260928 22:51:28.589401 29199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
I20260928 22:51:28.591324 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 29199
I20260928 22:51:28.591426 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance
I20260928 22:51:28.591701 16870 external_mini_cluster.cc:1428] Setting key 70ff7ca521f572a2a0dc5c22109e475c
Sep 28 22:51:28 dist-test-slave-56vw krb5kdc[28642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635888, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.121.130@KRBTEST.COM; defaulting to no policy
I20260928 22:51:28.604573 29332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:28.590161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:38111 (local address 127.16.121.129:54307)
0928 22:51:28.590384 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:28.590435 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 22:51:28.591130 (+ 695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:51:28.591445 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:51:28.591450 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:51:28.591726 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:51:28.592037 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:28.592045 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:28.592953 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:28.592954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:28.593800 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:28.593803 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:28.594242 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:28.594243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:28.594373 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:28.594964 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:51:28.594972 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:51:28.600334 (+ 5362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:51:28.602609 (+ 2275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:28.602623 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:28.602626 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:28.603106 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:28.603355 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:28.603355 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:28.603356 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:28.603510 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:28.603820 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:51:28.603827 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:51:28.604019 (+ 192us) client_negotiation.cc:769] Sending connection context
0928 22:51:28.604125 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 22:51:28.604304 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":89,"threads_started":1}
I20260928 22:51:28.605188 29333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:28.590231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:54307 (local address 127.16.121.190:38111)
0928 22:51:28.590382 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:28.590384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:28.590542 (+ 158us) server_negotiation.cc:407] Connection header received
0928 22:51:28.591252 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:28.591254 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:28.591270 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:28.591361 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:51:28.592142 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:28.592858 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:28.593912 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:28.594141 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:28.594223 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:28.600494 (+ 6271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:51:28.600496 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:51:28.600496 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:51:28.600507 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:51:28.602488 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:28.603206 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:28.603208 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:28.603208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:28.603282 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:28.603582 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:28.603583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:28.603583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:28.603736 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:51:28.603795 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:28.604931 (+ 1136us) server_negotiation.cc:299] Negotiation successful
0928 22:51:28.605054 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":76,"threads_started":1}
I20260928 22:51:28.605572 29330 heartbeater.cc:347] Connected to a master server at 127.16.121.190:38111
I20260928 22:51:28.605677 29330 heartbeater.cc:464] Registering TS with master...
I20260928 22:51:28.605886 29330 heartbeater.cc:511] Master 127.16.121.190:38111 requested a full tablet report, sending...
I20260928 22:51:28.606557 29118 ts_manager.cc:194] Registered new tserver with Master: 627f99588e0341d88d2272de309ddd6d (127.16.121.129:39337)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.130@KRBTEST.COM" created.
I20260928 22:51:28.607630 29118 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.129@KRBTEST.COM'} at 127.16.121.129:54307
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab.
Entry for principal kudu/127.16.121.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab.
I20260928 22:51:28.635566 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.121.148:51789/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.130:0
--local_ip_for_outbound_sockets=127.16.121.130
--webserver_interface=127.16.121.130
--webserver_port=0
--tserver_master_addrs=127.16.121.190:38111
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab
--principal=kudu/127.16.121.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:45883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 22:51:28 dist-test-slave-56vw krb5kdc[28642](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790635888, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 22:51:28.737005 29337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:51:28.737253 29337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:51:28.737294 29337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:51:28.738554 29337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:51:28.738631 29337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:51:28.738679 29337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.130
I20260928 22:51:28.740703 29337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:45883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.16.121.148:51789/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab
--principal=kudu/127.16.121.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.121.130
--webserver_port=0
--tserver_master_addrs=127.16.121.190:38111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.130
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:51:28.741156 29337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:51:28.741484 29337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:51:28.747829 29344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:51:28.748232 29346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:51:28.748313 29343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:51:28.749477 29337 server_base.cc:1034] running on GCE node
I20260928 22:51:28.749727 29337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:51:28.750057 29337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:51:28.751240 29337 hybrid_clock.cc:648] HybridClock initialized: now 1790635888751217 us; error 35 us; skew 500 ppm
Sep 28 22:51:28 dist-test-slave-56vw krb5kdc[28642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635888, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:51:28.754428 29337 init.cc:377] Logged in from keytab as kudu/127.16.121.130@KRBTEST.COM (short username kudu)
I20260928 22:51:28.754951 29337 webserver.cc:460] Webserver started at http://127.16.121.130:38037/ using document root <none> and password file <none>
I20260928 22:51:28.755281 29337 fs_manager.cc:359] Metadata directory not provided
I20260928 22:51:28.755338 29337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:51:28.755477 29337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 22:51:28 dist-test-slave-56vw krb5kdc[28642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635888, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.130@KRBTEST.COM for HTTP/127.16.121.148@KRBTEST.COM
I20260928 22:51:28.773366 29337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-1/data/instance:
uuid: "242d22b82213433a94d55a5104e65673"
format_stamp: "Formatted at 2026-09-28 22:51:28 on dist-test-slave-56vw"
server_key: "8324ee891e72303a3f9798a343b8a821"
server_key_iv: "e8de6dd4e69d0aae6b19cedabc523b71"
server_key_version: "kuduclusterkey@0"
I20260928 22:51:28.773914 29337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance:
uuid: "242d22b82213433a94d55a5104e65673"
format_stamp: "Formatted at 2026-09-28 22:51:28 on dist-test-slave-56vw"
server_key: "8324ee891e72303a3f9798a343b8a821"
server_key_iv: "e8de6dd4e69d0aae6b19cedabc523b71"
server_key_version: "kuduclusterkey@0"
I20260928 22:51:28.778620 29337 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 22:51:28.797119 29353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:28.797444 29337 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 22:51:28.797545 29337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-1/wal
uuid: "242d22b82213433a94d55a5104e65673"
format_stamp: "Formatted at 2026-09-28 22:51:28 on dist-test-slave-56vw"
server_key: "8324ee891e72303a3f9798a343b8a821"
server_key_iv: "e8de6dd4e69d0aae6b19cedabc523b71"
server_key_version: "kuduclusterkey@0"
I20260928 22:51:28.797647 29337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:51:28.815486 29337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:51:28.818058 29337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:51:28.818281 29337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:51:28.818594 29337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:51:28.819087 29337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:51:28.819177 29337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:28.819279 29337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:51:28.819341 29337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:28.826218 29337 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.130:43063
I20260928 22:51:28.826745 29337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
I20260928 22:51:28.835758 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 29337
I20260928 22:51:28.835891 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance
I20260928 22:51:28.836095 16870 external_mini_cluster.cc:1428] Setting key a90ec4a334581a1015bdb2896992820b
I20260928 22:51:28.836989 29466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.130:43063 every 8 connection(s)
WARNING: no policy specified for kudu/127.16.121.131@KRBTEST.COM; defaulting to no policy
Sep 28 22:51:28 dist-test-slave-56vw krb5kdc[28642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635888, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.130@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.131@KRBTEST.COM" created.
I20260928 22:51:28.859913 29333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:28.837989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:43283 (local address 127.16.121.190:38111)
0928 22:51:28.838039 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:28.838041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:28.838432 (+ 391us) server_negotiation.cc:407] Connection header received
0928 22:51:28.839196 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:28.839197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:28.839213 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:28.839244 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:51:28.840148 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:28.840960 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:28.848975 (+ 8015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:28.849376 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:28.849473 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:28.854993 (+ 5520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:51:28.854995 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:51:28.854995 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:51:28.855006 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:51:28.857844 (+ 2838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:28.858528 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:28.858530 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:28.858530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:28.858600 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:28.858970 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:28.858971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:28.858972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:28.859186 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:51:28.859248 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:28.859693 (+ 445us) server_negotiation.cc:299] Negotiation successful
0928 22:51:28.859802 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 22:51:28.860538 29470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:28.837677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:38111 (local address 127.16.121.130:43283)
0928 22:51:28.838286 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:28.838344 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 22:51:28.839037 (+ 693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:51:28.839444 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:51:28.839448 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:51:28.839732 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:51:28.840032 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:28.840039 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:28.841081 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:28.841081 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:28.848796 (+ 7715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:28.848802 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:28.849588 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:28.849590 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:28.849777 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:28.850471 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:51:28.850483 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:51:28.854816 (+ 4333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:51:28.857994 (+ 3178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:28.858005 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:28.858009 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:28.858411 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:28.858683 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:28.858684 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:28.858684 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:28.858864 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:28.859287 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:51:28.859294 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:51:28.859498 (+ 204us) client_negotiation.cc:769] Sending connection context
0928 22:51:28.860049 (+ 551us) client_negotiation.cc:240] Negotiation successful
0928 22:51:28.860256 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":103,"threads_started":1}
I20260928 22:51:28.861109 29467 heartbeater.cc:347] Connected to a master server at 127.16.121.190:38111
I20260928 22:51:28.861235 29467 heartbeater.cc:464] Registering TS with master...
I20260928 22:51:28.861471 29467 heartbeater.cc:511] Master 127.16.121.190:38111 requested a full tablet report, sending...
I20260928 22:51:28.861869 29118 ts_manager.cc:194] Registered new tserver with Master: 242d22b82213433a94d55a5104e65673 (127.16.121.130:43063)
I20260928 22:51:28.862480 29118 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.130@KRBTEST.COM'} at 127.16.121.130:43283
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab.
Entry for principal kudu/127.16.121.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab.
I20260928 22:51:28.882912 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.121.148:51789/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.131:0
--local_ip_for_outbound_sockets=127.16.121.131
--webserver_interface=127.16.121.131
--webserver_port=0
--tserver_master_addrs=127.16.121.190:38111
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab
--principal=kudu/127.16.121.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:45883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:51:29.007133 29473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:51:29.007436 29473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:51:29.007475 29473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:51:29.009320 29473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:51:29.009394 29473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:51:29.009436 29473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.131
I20260928 22:51:29.011888 29473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:45883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.16.121.148:51789/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab
--principal=kudu/127.16.121.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.121.131
--webserver_port=0
--tserver_master_addrs=127.16.121.190:38111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.131
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:51:29.012192 29473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:51:29.012490 29473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:51:29.019212 29478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:51:29.019227 29481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:51:29.020035 29473 server_base.cc:1034] running on GCE node
W20260928 22:51:29.019189 29479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:51:29.020517 29473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:51:29.020887 29473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:51:29.022120 29473 hybrid_clock.cc:648] HybridClock initialized: now 1790635889022091 us; error 44 us; skew 500 ppm
Sep 28 22:51:29 dist-test-slave-56vw krb5kdc[28642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635889, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:51:29.025700 29473 init.cc:377] Logged in from keytab as kudu/127.16.121.131@KRBTEST.COM (short username kudu)
I20260928 22:51:29.026294 29473 webserver.cc:460] Webserver started at http://127.16.121.131:38029/ using document root <none> and password file <none>
I20260928 22:51:29.026587 29473 fs_manager.cc:359] Metadata directory not provided
I20260928 22:51:29.026638 29473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:51:29.026774 29473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 22:51:29 dist-test-slave-56vw krb5kdc[28642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635889, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.131@KRBTEST.COM for HTTP/127.16.121.148@KRBTEST.COM
I20260928 22:51:29.045101 29473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-2/data/instance:
uuid: "f90087404db048ccbc1bb901612f59d4"
format_stamp: "Formatted at 2026-09-28 22:51:29 on dist-test-slave-56vw"
server_key: "849bfef581618e487cef9d1c2dd66918"
server_key_iv: "2012493050f74e5878d015f49c607164"
server_key_version: "kuduclusterkey@0"
I20260928 22:51:29.045732 29473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance:
uuid: "f90087404db048ccbc1bb901612f59d4"
format_stamp: "Formatted at 2026-09-28 22:51:29 on dist-test-slave-56vw"
server_key: "849bfef581618e487cef9d1c2dd66918"
server_key_iv: "2012493050f74e5878d015f49c607164"
server_key_version: "kuduclusterkey@0"
I20260928 22:51:29.051498 29473 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.000s
I20260928 22:51:29.067395 29488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:29.067628 29473 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 22:51:29.067749 29473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-2/wal
uuid: "f90087404db048ccbc1bb901612f59d4"
format_stamp: "Formatted at 2026-09-28 22:51:29 on dist-test-slave-56vw"
server_key: "849bfef581618e487cef9d1c2dd66918"
server_key_iv: "2012493050f74e5878d015f49c607164"
server_key_version: "kuduclusterkey@0"
I20260928 22:51:29.067859 29473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:51:29.092489 29473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:51:29.095446 29473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:51:29.095610 29473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:51:29.095835 29473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:51:29.096252 29473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:51:29.096303 29473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:29.096336 29473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:51:29.096365 29473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:29.103511 29473 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.131:46089
I20260928 22:51:29.103960 29473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
I20260928 22:51:29.111202 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 29473
I20260928 22:51:29.111333 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance
I20260928 22:51:29.111569 16870 external_mini_cluster.cc:1428] Setting key aeb1d4dfab4ba46256c5b73607fc4332
Sep 28 22:51:29 dist-test-slave-56vw krb5kdc[28642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635889, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.131@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:51:29.116477 29601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.131:46089 every 8 connection(s)
I20260928 22:51:29.117199 29333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:29.104929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:35479 (local address 127.16.121.190:38111)
0928 22:51:29.104981 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:29.104983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:29.105758 (+ 775us) server_negotiation.cc:407] Connection header received
0928 22:51:29.106448 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:29.106450 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:29.106463 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:29.106533 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:51:29.107327 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:29.108123 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:29.109670 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:29.109987 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:29.110034 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:29.113576 (+ 3542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:51:29.113577 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:51:29.113577 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:51:29.113586 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:51:29.115428 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:29.115953 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:29.115954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:29.115955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:29.116018 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:29.116351 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:29.116352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:29.116352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:29.116503 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:51:29.116549 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:29.116997 (+ 448us) server_negotiation.cc:299] Negotiation successful
0928 22:51:29.117105 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 22:51:29.117955 29604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:29.104902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:38111 (local address 127.16.121.131:35479)
0928 22:51:29.105591 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:29.105642 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 22:51:29.106313 (+ 671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:51:29.106612 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:51:29.106617 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:51:29.106872 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:51:29.107209 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:29.107216 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:29.108238 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:29.108240 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:29.109562 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:29.109565 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:29.110143 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:29.110144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:29.110295 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:29.111973 (+ 1678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:51:29.111985 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:51:29.113477 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:51:29.115525 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:29.115533 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:29.115536 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:29.115853 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:29.116109 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:29.116110 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:29.116110 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:29.116267 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:29.116666 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:51:29.116671 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:51:29.116831 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 22:51:29.117555 (+ 724us) client_negotiation.cc:240] Negotiation successful
0928 22:51:29.117727 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":580,"threads_started":1}
I20260928 22:51:29.118752 29602 heartbeater.cc:347] Connected to a master server at 127.16.121.190:38111
I20260928 22:51:29.118844 29602 heartbeater.cc:464] Registering TS with master...
I20260928 22:51:29.119023 29602 heartbeater.cc:511] Master 127.16.121.190:38111 requested a full tablet report, sending...
I20260928 22:51:29.119488 29118 ts_manager.cc:194] Registered new tserver with Master: f90087404db048ccbc1bb901612f59d4 (127.16.121.131:46089)
I20260928 22:51:29.120007 29118 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.131@KRBTEST.COM'} at 127.16.121.131:35479
I20260928 22:51:29.123852 16870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:51:29.124742 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 29199
I20260928 22:51:29.132511 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 29337
I20260928 22:51:29.137890 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 29473
I20260928 22:51:29.143473 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 29086
I20260928 22:51:29.150110 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:38111
--webserver_interface=127.16.121.190
--webserver_port=39329
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:45883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:38111
--ranger_config_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.121.148:51789/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.121.148:51789/kms with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:51:29.233698 29605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:51:29.233927 29605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:51:29.233959 29605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:51:29.235201 29605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:51:29.235268 29605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:51:29.235291 29605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:51:29.235322 29605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:51:29.235349 29605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:51:29.237083 29605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:45883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.121.148:51789/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:38111
--ranger_config_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:38111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.121.190
--webserver_port=39329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:51:29.237389 29605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:51:29.237628 29605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:51:29.243014 29610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:51:29.243002 29613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:51:29.243003 29611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:51:29.243034 29605 server_base.cc:1034] running on GCE node
I20260928 22:51:29.243583 29605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:51:29.243844 29605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:51:29.244999 29605 hybrid_clock.cc:648] HybridClock initialized: now 1790635889244987 us; error 40 us; skew 500 ppm
Sep 28 22:51:29 dist-test-slave-56vw krb5kdc[28642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635889, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:51:29.247881 29605 init.cc:377] Logged in from keytab as kudu/127.16.121.190@KRBTEST.COM (short username kudu)
I20260928 22:51:29.248376 29605 webserver.cc:460] Webserver started at http://127.16.121.190:39329/ using document root <none> and password file <none>
I20260928 22:51:29.248627 29605 fs_manager.cc:359] Metadata directory not provided
I20260928 22:51:29.248680 29605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 22:51:29 dist-test-slave-56vw krb5kdc[28642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635889, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for HTTP/127.16.121.148@KRBTEST.COM
I20260928 22:51:29.312453 29605 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:51:29.313321 29621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:29.313548 29605 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:51:29.313727 29605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "a50a70da076c47daa85d718844995547"
format_stamp: "Formatted at 2026-09-28 22:51:27 on dist-test-slave-56vw"
server_key: "0a81a4ad280c0d143e786aa79deb982b"
server_key_iv: "4d4a3c5aa1a68a1d99396937e679a2ec"
server_key_version: "kuduclusterkey@0"
I20260928 22:51:29.314134 29605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:51:29.331394 29605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:51:29.333137 29605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:51:29.333313 29605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:51:29.337785 29605 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.190:38111
I20260928 22:51:29.337829 29673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.190:38111 every 8 connection(s)
I20260928 22:51:29.338217 29605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:51:29.339650 29674 sys_catalog.cc:263] Verifying existing consensus state
I20260928 22:51:29.340472 29674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a50a70da076c47daa85d718844995547: Bootstrap starting.
I20260928 22:51:29.347219 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 29605
I20260928 22:51:29.347592 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.121.148:51789/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.129:39337
--local_ip_for_outbound_sockets=127.16.121.129
--tserver_master_addrs=127.16.121.190:38111
--webserver_port=41591
--webserver_interface=127.16.121.129
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab
--principal=kudu/127.16.121.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:45883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.121.148:51789/kms with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:51:29.349339 29674 log.cc:824] T 00000000000000000000000000000000 P a50a70da076c47daa85d718844995547: Log is configured to *not* fsync() on all Append() calls
I20260928 22:51:29.351533 29674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a50a70da076c47daa85d718844995547: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 22:51:29.351824 29674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a50a70da076c47daa85d718844995547: Bootstrap complete.
I20260928 22:51:29.354251 29674 raft_consensus.cc:348] T 00000000000000000000000000000000 P a50a70da076c47daa85d718844995547 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a50a70da076c47daa85d718844995547" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 38111 } }
I20260928 22:51:29.354518 29674 raft_consensus.cc:729] T 00000000000000000000000000000000 P a50a70da076c47daa85d718844995547 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a50a70da076c47daa85d718844995547, State: Initialized, Role: FOLLOWER
I20260928 22:51:29.354719 29674 consensus_queue.cc:260] T 00000000000000000000000000000000 P a50a70da076c47daa85d718844995547 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a50a70da076c47daa85d718844995547" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 38111 } }
I20260928 22:51:29.354817 29674 raft_consensus.cc:388] T 00000000000000000000000000000000 P a50a70da076c47daa85d718844995547 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:51:29.354869 29674 raft_consensus.cc:482] T 00000000000000000000000000000000 P a50a70da076c47daa85d718844995547 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:51:29.354916 29674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a50a70da076c47daa85d718844995547 [term 1 FOLLOWER]: Advancing to term 2
I20260928 22:51:29.356004 29674 raft_consensus.cc:504] T 00000000000000000000000000000000 P a50a70da076c47daa85d718844995547 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a50a70da076c47daa85d718844995547" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 38111 } }
I20260928 22:51:29.356139 29674 leader_election.cc:302] T 00000000000000000000000000000000 P a50a70da076c47daa85d718844995547 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a50a70da076c47daa85d718844995547; no voters:
I20260928 22:51:29.356336 29674 leader_election.cc:288] T 00000000000000000000000000000000 P a50a70da076c47daa85d718844995547 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 22:51:29.356397 29678 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a50a70da076c47daa85d718844995547 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 22:51:29.356565 29678 raft_consensus.cc:686] T 00000000000000000000000000000000 P a50a70da076c47daa85d718844995547 [term 2 LEADER]: Becoming Leader. State: Replica: a50a70da076c47daa85d718844995547, State: Running, Role: LEADER
I20260928 22:51:29.356604 29674 sys_catalog.cc:564] T 00000000000000000000000000000000 P a50a70da076c47daa85d718844995547 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:51:29.356683 29678 consensus_queue.cc:237] T 00000000000000000000000000000000 P a50a70da076c47daa85d718844995547 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a50a70da076c47daa85d718844995547" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 38111 } }
I20260928 22:51:29.356972 29678 sys_catalog.cc:455] T 00000000000000000000000000000000 P a50a70da076c47daa85d718844995547 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a50a70da076c47daa85d718844995547. Latest consensus state: current_term: 2 leader_uuid: "a50a70da076c47daa85d718844995547" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a50a70da076c47daa85d718844995547" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 38111 } } }
I20260928 22:51:29.357055 29678 sys_catalog.cc:458] T 00000000000000000000000000000000 P a50a70da076c47daa85d718844995547 [sys.catalog]: This master's current role is: LEADER
I20260928 22:51:29.357246 29679 sys_catalog.cc:455] T 00000000000000000000000000000000 P a50a70da076c47daa85d718844995547 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a50a70da076c47daa85d718844995547" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a50a70da076c47daa85d718844995547" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 38111 } } }
I20260928 22:51:29.357321 29679 sys_catalog.cc:458] T 00000000000000000000000000000000 P a50a70da076c47daa85d718844995547 [sys.catalog]: This master's current role is: LEADER
I20260928 22:51:29.357555 29684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:51:29.358089 29684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:51:29.358255 29684 catalog_manager.cc:1259] Loaded cluster ID: 4ddeb9ca1c7f42bab3f55ce382a93677
I20260928 22:51:29.358301 29684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:51:29.358443 29674 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 22:51:29.362063 29684 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:51:29.362584 29684 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a50a70da076c47daa85d718844995547: Loaded TSK: 0
I20260928 22:51:29.362852 29684 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 22:51:29.470376 29676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:51:29.470685 29676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:51:29.470729 29676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:51:29.472788 29676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:51:29.472862 29676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:51:29.472916 29676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.129
I20260928 22:51:29.475340 29676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:45883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.121.148:51789/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab
--principal=kudu/127.16.121.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.129:39337
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.121.129
--webserver_port=41591
--tserver_master_addrs=127.16.121.190:38111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.129
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:51:29.475626 29676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:51:29.475927 29676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:51:29.482534 29714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:51:29.482534 29711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:51:29.482975 29712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:51:29.483642 29676 server_base.cc:1034] running on GCE node
I20260928 22:51:29.483886 29676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:51:29.484190 29676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:51:29.485386 29676 hybrid_clock.cc:648] HybridClock initialized: now 1790635889485355 us; error 45 us; skew 500 ppm
Sep 28 22:51:29 dist-test-slave-56vw krb5kdc[28642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635889, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:51:29.493287 29676 init.cc:377] Logged in from keytab as kudu/127.16.121.129@KRBTEST.COM (short username kudu)
I20260928 22:51:29.493963 29676 webserver.cc:460] Webserver started at http://127.16.121.129:41591/ using document root <none> and password file <none>
I20260928 22:51:29.494266 29676 fs_manager.cc:359] Metadata directory not provided
I20260928 22:51:29.494341 29676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 22:51:29 dist-test-slave-56vw krb5kdc[28642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635889, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for HTTP/127.16.121.148@KRBTEST.COM
I20260928 22:51:29.512735 29676 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:51:29.513885 29722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:29.514117 29676 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:51:29.514209 29676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-0/wal
uuid: "627f99588e0341d88d2272de309ddd6d"
format_stamp: "Formatted at 2026-09-28 22:51:28 on dist-test-slave-56vw"
server_key: "5ad5568f0bdf58888af676083ab46d76"
server_key_iv: "bc8c98d5a9ec58fd4582612c410cddc0"
server_key_version: "kuduclusterkey@0"
I20260928 22:51:29.514662 29676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:51:29.525767 29676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:51:29.528076 29676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:51:29.528223 29676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:51:29.528472 29676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:51:29.528798 29676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:51:29.528852 29676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:29.528887 29676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:51:29.528915 29676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:29.534641 29676 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.129:39337
I20260928 22:51:29.534799 29835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.129:39337 every 8 connection(s)
I20260928 22:51:29.535149 29676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
I20260928 22:51:29.544327 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 29676
I20260928 22:51:29.544658 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.121.148:51789/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.130:43063
--local_ip_for_outbound_sockets=127.16.121.130
--tserver_master_addrs=127.16.121.190:38111
--webserver_port=38037
--webserver_interface=127.16.121.130
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab
--principal=kudu/127.16.121.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:45883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.121.148:51789/kms with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 22:51:29 dist-test-slave-56vw krb5kdc[28642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635889, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:51:29.550226 29839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:29.536013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:49963 (local address 127.16.121.190:38111)
0928 22:51:29.539222 (+ 3209us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:29.539233 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:29.539279 (+ 46us) server_negotiation.cc:407] Connection header received
0928 22:51:29.539441 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:29.539449 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:29.539510 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:29.539681 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:51:29.540489 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:29.541933 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:29.543072 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:29.543749 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:29.543875 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:29.546575 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:51:29.546580 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:51:29.546589 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:51:29.546602 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:51:29.548421 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:29.548907 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:29.548911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:29.548911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:29.549012 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:29.549247 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:29.549248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:29.549248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:29.549495 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:51:29.549593 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:29.549885 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 22:51:29.550011 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3136,"thread_start_us":70,"threads_started":1}
I20260928 22:51:29.550239 29838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:29.535873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:38111 (local address 127.16.121.129:49963)
0928 22:51:29.536077 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:29.536125 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 22:51:29.536770 (+ 645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:51:29.539757 (+ 2987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:51:29.539762 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:51:29.540027 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:51:29.540329 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:29.540337 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:29.542063 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:29.542064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:29.542950 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:29.542952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:29.543886 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:29.543887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:29.543995 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:29.544705 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:51:29.544717 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:51:29.546430 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:51:29.548533 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:29.548540 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:29.548542 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:29.548779 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:29.549072 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:29.549073 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:29.549073 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:29.549176 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:29.549575 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:51:29.549581 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:51:29.549724 (+ 143us) client_negotiation.cc:769] Sending connection context
0928 22:51:29.549834 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 22:51:29.550004 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":103,"threads_started":1}
I20260928 22:51:29.551602 29836 heartbeater.cc:347] Connected to a master server at 127.16.121.190:38111
I20260928 22:51:29.551716 29836 heartbeater.cc:464] Registering TS with master...
I20260928 22:51:29.551925 29836 heartbeater.cc:511] Master 127.16.121.190:38111 requested a full tablet report, sending...
W20260928 22:51:29.552822 29836 heartbeater.cc:644] Failed to heartbeat to 127.16.121.190:38111 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 22:51:29.552897 29836 heartbeater.cc:464] Registering TS with master...
I20260928 22:51:29.552968 29836 heartbeater.cc:511] Master 127.16.121.190:38111 requested a full tablet report, sending...
I20260928 22:51:29.553297 29836 heartbeater.cc:464] Registering TS with master...
I20260928 22:51:29.553390 29836 heartbeater.cc:511] Master 127.16.121.190:38111 requested a full tablet report, sending...
W20260928 22:51:29.553726 29836 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 22:51:29.670230 29840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:51:29.670542 29840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:51:29.670589 29840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:51:29.672472 29840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:51:29.672611 29840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:51:29.672668 29840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.130
I20260928 22:51:29.674978 29840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:45883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.121.148:51789/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab
--principal=kudu/127.16.121.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.130:43063
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.121.130
--webserver_port=38037
--tserver_master_addrs=127.16.121.190:38111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.130
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:51:29.675289 29840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:51:29.675576 29840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:51:29.682647 29846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:51:29.682862 29848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:51:29.682797 29845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:51:29.683226 29840 server_base.cc:1034] running on GCE node
I20260928 22:51:29.683435 29840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:51:29.683745 29840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:51:29.684933 29840 hybrid_clock.cc:648] HybridClock initialized: now 1790635889684900 us; error 44 us; skew 500 ppm
Sep 28 22:51:29 dist-test-slave-56vw krb5kdc[28642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635889, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:51:29.688268 29840 init.cc:377] Logged in from keytab as kudu/127.16.121.130@KRBTEST.COM (short username kudu)
I20260928 22:51:29.688886 29840 webserver.cc:460] Webserver started at http://127.16.121.130:38037/ using document root <none> and password file <none>
I20260928 22:51:29.689177 29840 fs_manager.cc:359] Metadata directory not provided
I20260928 22:51:29.689260 29840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 22:51:29 dist-test-slave-56vw krb5kdc[28642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635889, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.130@KRBTEST.COM for HTTP/127.16.121.148@KRBTEST.COM
I20260928 22:51:29.705303 29840 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:51:29.706293 29856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:29.706542 29840 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:51:29.706648 29840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-1/wal
uuid: "242d22b82213433a94d55a5104e65673"
format_stamp: "Formatted at 2026-09-28 22:51:28 on dist-test-slave-56vw"
server_key: "8324ee891e72303a3f9798a343b8a821"
server_key_iv: "e8de6dd4e69d0aae6b19cedabc523b71"
server_key_version: "kuduclusterkey@0"
I20260928 22:51:29.707116 29840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:51:29.723409 29840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:51:29.725867 29840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:51:29.726092 29840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:51:29.726394 29840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:51:29.726811 29840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:51:29.726874 29840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:29.726912 29840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:51:29.726931 29840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:29.734516 29840 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.130:43063
I20260928 22:51:29.734628 29969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.130:43063 every 8 connection(s)
I20260928 22:51:29.735019 29840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
I20260928 22:51:29.742074 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 29840
I20260928 22:51:29.742395 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.121.148:51789/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.131:46089
--local_ip_for_outbound_sockets=127.16.121.131
--tserver_master_addrs=127.16.121.190:38111
--webserver_port=38029
--webserver_interface=127.16.121.131
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab
--principal=kudu/127.16.121.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:45883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.121.148:51789/kms with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 22:51:29 dist-test-slave-56vw krb5kdc[28642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635889, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.130@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:51:29.747018 29839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:29.736227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:56309 (local address 127.16.121.190:38111)
0928 22:51:29.736290 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:29.736292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:29.736306 (+ 14us) server_negotiation.cc:407] Connection header received
0928 22:51:29.737035 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:29.737037 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:29.737053 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:29.737120 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:51:29.737881 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:29.738667 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:29.740261 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:29.740545 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:29.740596 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:29.743296 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:51:29.743297 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:51:29.743298 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:51:29.743306 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:51:29.745149 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:29.745720 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:29.745722 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:29.745722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:29.745786 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:29.746115 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:29.746116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:29.746117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:29.746287 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:51:29.746333 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:29.746765 (+ 432us) server_negotiation.cc:299] Negotiation successful
0928 22:51:29.746869 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 22:51:29.747695 29972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:29.735904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:38111 (local address 127.16.121.130:56309)
0928 22:51:29.736119 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:29.736171 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 22:51:29.736902 (+ 731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:51:29.737214 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:51:29.737218 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:51:29.737457 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:51:29.737764 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:29.737771 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:29.738764 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:29.738765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:29.740147 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:29.740150 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:29.740728 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:29.740729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:29.740883 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:29.741518 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:51:29.741529 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:51:29.743173 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:51:29.745257 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:29.745266 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:29.745269 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:29.745603 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:29.745883 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:29.745883 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:29.745884 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:29.746039 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:29.746383 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:51:29.746390 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:51:29.746604 (+ 214us) client_negotiation.cc:769] Sending connection context
0928 22:51:29.747194 (+ 590us) client_negotiation.cc:240] Negotiation successful
0928 22:51:29.747390 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":96,"threads_started":1}
I20260928 22:51:29.748284 29970 heartbeater.cc:347] Connected to a master server at 127.16.121.190:38111
I20260928 22:51:29.748394 29970 heartbeater.cc:464] Registering TS with master...
I20260928 22:51:29.748621 29970 heartbeater.cc:511] Master 127.16.121.190:38111 requested a full tablet report, sending...
W20260928 22:51:29.749298 29970 heartbeater.cc:644] Failed to heartbeat to 127.16.121.190:38111 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 22:51:29.749382 29970 heartbeater.cc:464] Registering TS with master...
I20260928 22:51:29.749451 29970 heartbeater.cc:511] Master 127.16.121.190:38111 requested a full tablet report, sending...
I20260928 22:51:29.749776 29970 heartbeater.cc:464] Registering TS with master...
I20260928 22:51:29.749864 29970 heartbeater.cc:511] Master 127.16.121.190:38111 requested a full tablet report, sending...
W20260928 22:51:29.750176 29970 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 22:51:29.856040 29973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:51:29.856345 29973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:51:29.856384 29973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:51:29.858495 29973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:51:29.858675 29973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:51:29.858726 29973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.131
I20260928 22:51:29.861232 29973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:45883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.121.148:51789/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab
--principal=kudu/127.16.121.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.131:46089
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.121.131
--webserver_port=38029
--tserver_master_addrs=127.16.121.190:38111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.131
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:51:29.861536 29973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:51:29.861822 29973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:51:29.868477 29982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:51:29.868593 29980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:51:29.868794 29973 server_base.cc:1034] running on GCE node
W20260928 22:51:29.869004 29979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:51:29.869244 29973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:51:29.869540 29973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:51:29.870711 29973 hybrid_clock.cc:648] HybridClock initialized: now 1790635889870682 us; error 44 us; skew 500 ppm
Sep 28 22:51:29 dist-test-slave-56vw krb5kdc[28642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635889, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:51:29.874289 29973 init.cc:377] Logged in from keytab as kudu/127.16.121.131@KRBTEST.COM (short username kudu)
I20260928 22:51:29.874831 29973 webserver.cc:460] Webserver started at http://127.16.121.131:38029/ using document root <none> and password file <none>
I20260928 22:51:29.875142 29973 fs_manager.cc:359] Metadata directory not provided
I20260928 22:51:29.875226 29973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 22:51:29 dist-test-slave-56vw krb5kdc[28642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635889, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.131@KRBTEST.COM for HTTP/127.16.121.148@KRBTEST.COM
I20260928 22:51:29.895782 29973 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.000s
I20260928 22:51:29.896932 29990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:29.897172 29973 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 22:51:29.897261 29973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-2/wal
uuid: "f90087404db048ccbc1bb901612f59d4"
format_stamp: "Formatted at 2026-09-28 22:51:29 on dist-test-slave-56vw"
server_key: "849bfef581618e487cef9d1c2dd66918"
server_key_iv: "2012493050f74e5878d015f49c607164"
server_key_version: "kuduclusterkey@0"
I20260928 22:51:29.897750 29973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:51:29.910542 29973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:51:29.912899 29973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:51:29.913079 29973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:51:29.913362 29973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:51:29.913760 29973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:51:29.913815 29973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:29.913856 29973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:51:29.913882 29973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:29.921577 29973 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.131:46089
I20260928 22:51:29.922219 29973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
I20260928 22:51:29.923537 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 29973
I20260928 22:51:29.923789 30103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.131:46089 every 8 connection(s)
Sep 28 22:51:29 dist-test-slave-56vw krb5kdc[28642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635889, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.131@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:51:29.938129 29839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:29.924765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:58565 (local address 127.16.121.190:38111)
0928 22:51:29.924825 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:29.924827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:29.924840 (+ 13us) server_negotiation.cc:407] Connection header received
0928 22:51:29.925567 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:29.925569 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:29.925583 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:29.925618 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:51:29.926563 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:29.927347 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:29.930353 (+ 3006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:29.930697 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:29.930756 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:29.934664 (+ 3908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:51:29.934666 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:51:29.934666 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:51:29.934675 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:51:29.936337 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:29.936914 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:29.936915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:29.936916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:29.936978 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:29.937286 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:29.937287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:29.937287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:29.937442 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:51:29.937484 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:29.937894 (+ 410us) server_negotiation.cc:299] Negotiation successful
0928 22:51:29.937985 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 22:51:29.938977 30106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:29.923660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:38111 (local address 127.16.121.131:58565)
0928 22:51:29.924392 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:29.924452 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 22:51:29.925305 (+ 853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:51:29.925835 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:51:29.925839 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:51:29.926089 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:51:29.926417 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:29.926424 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:29.927478 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:29.927479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:29.929970 (+ 2491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:29.929977 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:29.930888 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:29.930889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:29.931098 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:29.932506 (+ 1408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:51:29.932519 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:51:29.934489 (+ 1970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:51:29.936443 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:29.936454 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:29.936458 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:29.936814 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:29.937049 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:29.937049 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:29.937050 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:29.937212 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:29.937532 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:51:29.937539 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:51:29.937730 (+ 191us) client_negotiation.cc:769] Sending connection context
0928 22:51:29.938462 (+ 732us) client_negotiation.cc:240] Negotiation successful
0928 22:51:29.938678 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"thread_start_us":95,"threads_started":1}
I20260928 22:51:29.939764 30104 heartbeater.cc:347] Connected to a master server at 127.16.121.190:38111
I20260928 22:51:29.939883 30104 heartbeater.cc:464] Registering TS with master...
I20260928 22:51:29.940111 30104 heartbeater.cc:511] Master 127.16.121.190:38111 requested a full tablet report, sending...
W20260928 22:51:29.940917 30104 heartbeater.cc:644] Failed to heartbeat to 127.16.121.190:38111 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 22:51:29.940999 30104 heartbeater.cc:464] Registering TS with master...
I20260928 22:51:29.941071 30104 heartbeater.cc:511] Master 127.16.121.190:38111 requested a full tablet report, sending...
I20260928 22:51:29.941522 30104 heartbeater.cc:464] Registering TS with master...
I20260928 22:51:29.941650 30104 heartbeater.cc:511] Master 127.16.121.190:38111 requested a full tablet report, sending...
W20260928 22:51:29.941989 30104 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 22:51:29.942472 30108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:29.929297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51918 (local address 127.16.121.190:38111)
0928 22:51:29.931444 (+ 2147us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:29.931446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:29.931455 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:51:29.931479 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:29.931479 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:29.931493 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:29.933233 (+ 1740us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:51:29.933282 (+ 49us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:29.935178 (+ 1896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:29.937406 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:29.937717 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:29.939313 (+ 1596us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:29.939360 (+ 47us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:51:29.939361 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:51:29.939361 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:51:29.939365 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:51:29.941275 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:29.941617 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:29.941618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:29.941619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:29.941677 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:29.941903 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:29.941904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:29.941905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:29.942065 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:51:29.942245 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:29.942283 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 22:51:29.942364 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2111,"thread_start_us":132,"threads_started":1}
Sep 28 22:51:30 dist-test-slave-56vw krb5kdc[28642](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790635890, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:51:30.554069 29836 heartbeater.cc:464] Registering TS with master...
I20260928 22:51:30.554231 29836 heartbeater.cc:511] Master 127.16.121.190:38111 requested a full tablet report, sending...
I20260928 22:51:30.554685 29638 ts_manager.cc:194] Registered new tserver with Master: 627f99588e0341d88d2272de309ddd6d (127.16.121.129:39337)
I20260928 22:51:30.555838 29638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.129@KRBTEST.COM'} at 127.16.121.129:49963
I20260928 22:51:30.750383 29970 heartbeater.cc:464] Registering TS with master...
I20260928 22:51:30.750546 29970 heartbeater.cc:511] Master 127.16.121.190:38111 requested a full tablet report, sending...
I20260928 22:51:30.750983 29638 ts_manager.cc:194] Registered new tserver with Master: 242d22b82213433a94d55a5104e65673 (127.16.121.130:43063)
I20260928 22:51:30.751590 29638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.130@KRBTEST.COM'} at 127.16.121.130:56309
I20260928 22:51:30.942173 30104 heartbeater.cc:464] Registering TS with master...
I20260928 22:51:30.942332 30104 heartbeater.cc:511] Master 127.16.121.190:38111 requested a full tablet report, sending...
I20260928 22:51:30.942728 29638 ts_manager.cc:194] Registered new tserver with Master: f90087404db048ccbc1bb901612f59d4 (127.16.121.131:46089)
I20260928 22:51:30.943393 29638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.131@KRBTEST.COM'} at 127.16.121.131:58565
I20260928 22:51:30.950963 16870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:51:30.961918 30125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:30.952692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51926 (local address 127.16.121.190:38111)
0928 22:51:30.953020 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:30.953022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:30.953208 (+ 186us) server_negotiation.cc:407] Connection header received
0928 22:51:30.953237 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:30.953238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:30.953255 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:30.953312 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:51:30.954030 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:30.955014 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:30.956145 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:30.956495 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:30.956561 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:30.958390 (+ 1829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:51:30.958391 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:51:30.958392 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:51:30.958400 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:51:30.960229 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:30.960661 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:30.960663 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:30.960663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:30.960726 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:30.960947 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:30.960948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:30.960949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:30.961094 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:51:30.961207 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:30.961713 (+ 506us) server_negotiation.cc:299] Negotiation successful
0928 22:51:30.961805 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":74,"threads_started":1}
Sep 28 22:51:30 dist-test-slave-56vw krb5kdc[28642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635890, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for HTTP/127.16.121.148@KRBTEST.COM
I20260928 22:51:30.964263 29638 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51926:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:51:30.966820 29638 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:51:30.986064 30134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:30.978577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.131:46089 (local address 127.0.0.1:60962)
0928 22:51:30.980885 (+ 2308us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:30.980937 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 22:51:30.981007 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:51:30.981859 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:51:30.981863 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:51:30.981865 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:51:30.981943 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:30.981944 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:30.983161 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:30.983162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:30.984719 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:30.984723 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:30.985740 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:30.985741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:30.985825 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:30.985839 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 22:51:30.985908 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 22:51:30.985972 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2269,"thread_start_us":87,"threads_started":1}
I20260928 22:51:30.986935 30132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:30.978841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60962 (local address 127.16.121.131:46089)
0928 22:51:30.981352 (+ 2511us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:30.981358 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:30.981368 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:51:30.981403 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:30.981405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:30.981421 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:30.981486 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:51:30.982020 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:30.983040 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:30.984817 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:30.985571 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:30.985645 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:30.985705 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:30.986535 (+ 830us) server_negotiation.cc:299] Negotiation successful
0928 22:51:30.986584 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2472,"thread_start_us":46,"threads_started":1}
I20260928 22:51:30.987871 30038 tablet_service.cc:1467] Processing CreateTablet for tablet 6976ffc957b64b31b24d210773f62a58 (DEFAULT_TABLE table=test-table [id=ce81de63c0a34c52983509afb94b2eb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:51:30.988262 30038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6976ffc957b64b31b24d210773f62a58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:51:31.000025 30133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:30.979016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:39337 (local address 127.0.0.1:54224)
0928 22:51:30.981116 (+ 2100us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:30.981123 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 22:51:30.981155 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:51:30.992972 (+ 11817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:51:30.992973 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:51:30.992976 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:51:30.993048 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:30.993051 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:30.995822 (+ 2771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:30.995823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:30.996927 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:30.996931 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:30.999795 (+ 2864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:30.999796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:30.999854 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:30.999859 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:51:30.999886 (+ 27us) client_negotiation.cc:240] Negotiation successful
0928 22:51:30.999934 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2076,"thread_start_us":102,"threads_started":1}
I20260928 22:51:31.001196 30135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:30.979490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54224 (local address 127.16.121.129:39337)
0928 22:51:30.981352 (+ 1862us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:30.981358 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:30.992790 (+ 11432us) server_negotiation.cc:407] Connection header received
0928 22:51:30.992836 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:30.992839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:30.992857 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:30.993140 (+ 283us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:51:30.993236 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:30.994354 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:30.997036 (+ 2682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:30.997785 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:31.000941 (+ 3156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:31.001037 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:31.001065 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 22:51:31.001114 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1819,"thread_start_us":70,"threads_started":1}
I20260928 22:51:31.002332 29770 tablet_service.cc:1467] Processing CreateTablet for tablet 6976ffc957b64b31b24d210773f62a58 (DEFAULT_TABLE table=test-table [id=ce81de63c0a34c52983509afb94b2eb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:51:31.002745 29770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6976ffc957b64b31b24d210773f62a58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:51:31.005275 30137 tablet_bootstrap.cc:492] T 6976ffc957b64b31b24d210773f62a58 P f90087404db048ccbc1bb901612f59d4: Bootstrap starting.
I20260928 22:51:31.006021 30137 tablet_bootstrap.cc:654] T 6976ffc957b64b31b24d210773f62a58 P f90087404db048ccbc1bb901612f59d4: Neither blocks nor log segments found. Creating new log.
I20260928 22:51:31.006330 30137 log.cc:824] T 6976ffc957b64b31b24d210773f62a58 P f90087404db048ccbc1bb901612f59d4: Log is configured to *not* fsync() on all Append() calls
I20260928 22:51:31.008033 30131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:30.979207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.130:43063 (local address 127.0.0.1:49592)
0928 22:51:30.981213 (+ 2006us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:30.981220 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 22:51:30.981249 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:51:30.994709 (+ 13460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:51:30.994710 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:51:30.994713 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:51:30.994765 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:30.994767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:30.998568 (+ 3801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:30.998569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:30.999681 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:30.999684 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:31.007784 (+ 8100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:31.007786 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:31.007867 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:31.007874 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 22:51:31.007902 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 22:51:31.007955 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1997,"thread_start_us":48,"threads_started":1}
I20260928 22:51:31.008731 30136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:30.979639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49592 (local address 127.16.121.130:43063)
0928 22:51:30.994529 (+ 14890us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:30.994534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:30.994545 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:51:30.994581 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:30.994583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:30.994600 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:30.994838 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:51:30.994932 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:30.998456 (+ 3524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:31.000453 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:31.007670 (+ 7217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:31.008486 (+ 816us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:31.008581 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:31.008611 (+ 30us) server_negotiation.cc:299] Negotiation successful
0928 22:51:31.008656 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14834,"thread_start_us":50,"threads_started":1}
I20260928 22:51:31.009567 29904 tablet_service.cc:1467] Processing CreateTablet for tablet 6976ffc957b64b31b24d210773f62a58 (DEFAULT_TABLE table=test-table [id=ce81de63c0a34c52983509afb94b2eb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:51:31.009944 29904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6976ffc957b64b31b24d210773f62a58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:51:31.014101 30138 tablet_bootstrap.cc:492] T 6976ffc957b64b31b24d210773f62a58 P 627f99588e0341d88d2272de309ddd6d: Bootstrap starting.
I20260928 22:51:31.014837 30138 tablet_bootstrap.cc:654] T 6976ffc957b64b31b24d210773f62a58 P 627f99588e0341d88d2272de309ddd6d: Neither blocks nor log segments found. Creating new log.
I20260928 22:51:31.015179 30138 log.cc:824] T 6976ffc957b64b31b24d210773f62a58 P 627f99588e0341d88d2272de309ddd6d: Log is configured to *not* fsync() on all Append() calls
I20260928 22:51:31.015829 30140 tablet_bootstrap.cc:492] T 6976ffc957b64b31b24d210773f62a58 P 242d22b82213433a94d55a5104e65673: Bootstrap starting.
I20260928 22:51:31.016134 30138 tablet_bootstrap.cc:492] T 6976ffc957b64b31b24d210773f62a58 P 627f99588e0341d88d2272de309ddd6d: No bootstrap required, opened a new log
I20260928 22:51:31.016274 30138 ts_tablet_manager.cc:1397] T 6976ffc957b64b31b24d210773f62a58 P 627f99588e0341d88d2272de309ddd6d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:51:31.016433 30140 tablet_bootstrap.cc:654] T 6976ffc957b64b31b24d210773f62a58 P 242d22b82213433a94d55a5104e65673: Neither blocks nor log segments found. Creating new log.
I20260928 22:51:31.016825 30140 log.cc:824] T 6976ffc957b64b31b24d210773f62a58 P 242d22b82213433a94d55a5104e65673: Log is configured to *not* fsync() on all Append() calls
I20260928 22:51:31.017426 30137 tablet_bootstrap.cc:492] T 6976ffc957b64b31b24d210773f62a58 P f90087404db048ccbc1bb901612f59d4: No bootstrap required, opened a new log
I20260928 22:51:31.017508 30137 ts_tablet_manager.cc:1397] T 6976ffc957b64b31b24d210773f62a58 P f90087404db048ccbc1bb901612f59d4: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20260928 22:51:31.019510 30138 raft_consensus.cc:348] T 6976ffc957b64b31b24d210773f62a58 P 627f99588e0341d88d2272de309ddd6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f90087404db048ccbc1bb901612f59d4" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 46089 } } peers { permanent_uuid: "627f99588e0341d88d2272de309ddd6d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 39337 } } peers { permanent_uuid: "242d22b82213433a94d55a5104e65673" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 43063 } }
I20260928 22:51:31.019820 30138 raft_consensus.cc:374] T 6976ffc957b64b31b24d210773f62a58 P 627f99588e0341d88d2272de309ddd6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:51:31.019918 30138 raft_consensus.cc:729] T 6976ffc957b64b31b24d210773f62a58 P 627f99588e0341d88d2272de309ddd6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 627f99588e0341d88d2272de309ddd6d, State: Initialized, Role: FOLLOWER
I20260928 22:51:31.020186 30138 consensus_queue.cc:260] T 6976ffc957b64b31b24d210773f62a58 P 627f99588e0341d88d2272de309ddd6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f90087404db048ccbc1bb901612f59d4" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 46089 } } peers { permanent_uuid: "627f99588e0341d88d2272de309ddd6d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 39337 } } peers { permanent_uuid: "242d22b82213433a94d55a5104e65673" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 43063 } }
I20260928 22:51:31.020582 30138 ts_tablet_manager.cc:1428] T 6976ffc957b64b31b24d210773f62a58 P 627f99588e0341d88d2272de309ddd6d: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260928 22:51:31.020759 29836 heartbeater.cc:503] Master 127.16.121.190:38111 was elected leader, sending a full tablet report...
I20260928 22:51:31.020665 30137 raft_consensus.cc:348] T 6976ffc957b64b31b24d210773f62a58 P f90087404db048ccbc1bb901612f59d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f90087404db048ccbc1bb901612f59d4" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 46089 } } peers { permanent_uuid: "627f99588e0341d88d2272de309ddd6d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 39337 } } peers { permanent_uuid: "242d22b82213433a94d55a5104e65673" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 43063 } }
I20260928 22:51:31.020826 30137 raft_consensus.cc:374] T 6976ffc957b64b31b24d210773f62a58 P f90087404db048ccbc1bb901612f59d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:51:31.020857 30137 raft_consensus.cc:729] T 6976ffc957b64b31b24d210773f62a58 P f90087404db048ccbc1bb901612f59d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f90087404db048ccbc1bb901612f59d4, State: Initialized, Role: FOLLOWER
I20260928 22:51:31.020962 30137 consensus_queue.cc:260] T 6976ffc957b64b31b24d210773f62a58 P f90087404db048ccbc1bb901612f59d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f90087404db048ccbc1bb901612f59d4" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 46089 } } peers { permanent_uuid: "627f99588e0341d88d2272de309ddd6d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 39337 } } peers { permanent_uuid: "242d22b82213433a94d55a5104e65673" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 43063 } }
I20260928 22:51:31.021222 30137 ts_tablet_manager.cc:1428] T 6976ffc957b64b31b24d210773f62a58 P f90087404db048ccbc1bb901612f59d4: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 22:51:31.021667 30104 heartbeater.cc:503] Master 127.16.121.190:38111 was elected leader, sending a full tablet report...
I20260928 22:51:31.023844 30140 tablet_bootstrap.cc:492] T 6976ffc957b64b31b24d210773f62a58 P 242d22b82213433a94d55a5104e65673: No bootstrap required, opened a new log
I20260928 22:51:31.023972 30140 ts_tablet_manager.cc:1397] T 6976ffc957b64b31b24d210773f62a58 P 242d22b82213433a94d55a5104e65673: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260928 22:51:31.026983 30140 raft_consensus.cc:348] T 6976ffc957b64b31b24d210773f62a58 P 242d22b82213433a94d55a5104e65673 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f90087404db048ccbc1bb901612f59d4" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 46089 } } peers { permanent_uuid: "627f99588e0341d88d2272de309ddd6d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 39337 } } peers { permanent_uuid: "242d22b82213433a94d55a5104e65673" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 43063 } }
I20260928 22:51:31.027451 30140 raft_consensus.cc:374] T 6976ffc957b64b31b24d210773f62a58 P 242d22b82213433a94d55a5104e65673 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:51:31.027540 30140 raft_consensus.cc:729] T 6976ffc957b64b31b24d210773f62a58 P 242d22b82213433a94d55a5104e65673 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 242d22b82213433a94d55a5104e65673, State: Initialized, Role: FOLLOWER
I20260928 22:51:31.027719 30140 consensus_queue.cc:260] T 6976ffc957b64b31b24d210773f62a58 P 242d22b82213433a94d55a5104e65673 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f90087404db048ccbc1bb901612f59d4" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 46089 } } peers { permanent_uuid: "627f99588e0341d88d2272de309ddd6d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 39337 } } peers { permanent_uuid: "242d22b82213433a94d55a5104e65673" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 43063 } }
I20260928 22:51:31.028051 30140 ts_tablet_manager.cc:1428] T 6976ffc957b64b31b24d210773f62a58 P 242d22b82213433a94d55a5104e65673: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 22:51:31.028204 29970 heartbeater.cc:503] Master 127.16.121.190:38111 was elected leader, sending a full tablet report...
W20260928 22:51:31.036952 29837 tablet.cc:2369] T 6976ffc957b64b31b24d210773f62a58 P 627f99588e0341d88d2272de309ddd6d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:51:31.160669 30143 raft_consensus.cc:482] T 6976ffc957b64b31b24d210773f62a58 P 627f99588e0341d88d2272de309ddd6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:51:31.160835 30143 raft_consensus.cc:504] T 6976ffc957b64b31b24d210773f62a58 P 627f99588e0341d88d2272de309ddd6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f90087404db048ccbc1bb901612f59d4" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 46089 } } peers { permanent_uuid: "627f99588e0341d88d2272de309ddd6d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 39337 } } peers { permanent_uuid: "242d22b82213433a94d55a5104e65673" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 43063 } }
I20260928 22:51:31.161286 30143 leader_election.cc:288] T 6976ffc957b64b31b24d210773f62a58 P 627f99588e0341d88d2272de309ddd6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f90087404db048ccbc1bb901612f59d4 (127.16.121.131:46089), 242d22b82213433a94d55a5104e65673 (127.16.121.130:43063)
I20260928 22:51:31.166620 30136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:31.161718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:48749 (local address 127.16.121.130:43063)
0928 22:51:31.161764 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:31.161767 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:31.163330 (+ 1563us) server_negotiation.cc:407] Connection header received
0928 22:51:31.163426 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:31.163428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:31.163437 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:31.163465 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:51:31.163657 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:31.164327 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:31.165661 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:31.166209 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:31.166267 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:31.166276 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:31.166490 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 22:51:31.166550 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 22:51:31.166620 30132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:31.161667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:33221 (local address 127.16.121.131:46089)
0928 22:51:31.162934 (+ 1267us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:31.162936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:31.162946 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:51:31.162991 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:31.162992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:31.163003 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:31.163177 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:51:31.163414 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:31.164139 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:31.165613 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:31.166157 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:31.166226 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:31.166236 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:31.166485 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 22:51:31.166550 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1237}
I20260928 22:51:31.166941 30147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:31.161453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.131:46089 (local address 127.16.121.129:33221)
0928 22:51:31.161828 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:31.161840 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:51:31.161898 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:51:31.163210 (+ 1312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:51:31.163212 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:51:31.163215 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:51:31.163300 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:31.163302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:31.164260 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:31.164261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:31.165503 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:31.165506 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:31.166317 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:31.166318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:31.166391 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:31.166395 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:51:31.166797 (+ 402us) client_negotiation.cc:240] Negotiation successful
0928 22:51:31.166844 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":90,"threads_started":1}
I20260928 22:51:31.167354 30146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:31.161501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.130:43063 (local address 127.16.121.129:48749)
0928 22:51:31.163267 (+ 1766us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:31.163278 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:51:31.163356 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:51:31.163522 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:51:31.163523 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:51:31.163525 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:51:31.163589 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:31.163591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:31.164426 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:31.164427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:31.165552 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:31.165555 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:31.166334 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:31.166335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:31.166412 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:31.166415 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:51:31.166972 (+ 557us) client_negotiation.cc:240] Negotiation successful
0928 22:51:31.167275 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1747,"thread_start_us":100,"threads_started":1}
I20260928 22:51:31.167373 30047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6976ffc957b64b31b24d210773f62a58" candidate_uuid: "627f99588e0341d88d2272de309ddd6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f90087404db048ccbc1bb901612f59d4" is_pre_election: true
I20260928 22:51:31.167546 30047 raft_consensus.cc:2454] T 6976ffc957b64b31b24d210773f62a58 P f90087404db048ccbc1bb901612f59d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 627f99588e0341d88d2272de309ddd6d in term 0.
I20260928 22:51:31.167805 29726 leader_election.cc:302] T 6976ffc957b64b31b24d210773f62a58 P 627f99588e0341d88d2272de309ddd6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 627f99588e0341d88d2272de309ddd6d, f90087404db048ccbc1bb901612f59d4; no voters:
I20260928 22:51:31.167819 29924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6976ffc957b64b31b24d210773f62a58" candidate_uuid: "627f99588e0341d88d2272de309ddd6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "242d22b82213433a94d55a5104e65673" is_pre_election: true
I20260928 22:51:31.167936 30143 raft_consensus.cc:2781] T 6976ffc957b64b31b24d210773f62a58 P 627f99588e0341d88d2272de309ddd6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:51:31.167963 29924 raft_consensus.cc:2454] T 6976ffc957b64b31b24d210773f62a58 P 242d22b82213433a94d55a5104e65673 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 627f99588e0341d88d2272de309ddd6d in term 0.
I20260928 22:51:31.167979 30143 raft_consensus.cc:482] T 6976ffc957b64b31b24d210773f62a58 P 627f99588e0341d88d2272de309ddd6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:51:31.168001 30143 raft_consensus.cc:3037] T 6976ffc957b64b31b24d210773f62a58 P 627f99588e0341d88d2272de309ddd6d [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:51:31.168931 30143 raft_consensus.cc:504] T 6976ffc957b64b31b24d210773f62a58 P 627f99588e0341d88d2272de309ddd6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f90087404db048ccbc1bb901612f59d4" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 46089 } } peers { permanent_uuid: "627f99588e0341d88d2272de309ddd6d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 39337 } } peers { permanent_uuid: "242d22b82213433a94d55a5104e65673" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 43063 } }
I20260928 22:51:31.169070 30143 leader_election.cc:288] T 6976ffc957b64b31b24d210773f62a58 P 627f99588e0341d88d2272de309ddd6d [CANDIDATE]: Term 1 election: Requested vote from peers f90087404db048ccbc1bb901612f59d4 (127.16.121.131:46089), 242d22b82213433a94d55a5104e65673 (127.16.121.130:43063)
I20260928 22:51:31.169261 30047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6976ffc957b64b31b24d210773f62a58" candidate_uuid: "627f99588e0341d88d2272de309ddd6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f90087404db048ccbc1bb901612f59d4"
I20260928 22:51:31.169268 29924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6976ffc957b64b31b24d210773f62a58" candidate_uuid: "627f99588e0341d88d2272de309ddd6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "242d22b82213433a94d55a5104e65673"
I20260928 22:51:31.169337 29924 raft_consensus.cc:3037] T 6976ffc957b64b31b24d210773f62a58 P 242d22b82213433a94d55a5104e65673 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:51:31.169337 30047 raft_consensus.cc:3037] T 6976ffc957b64b31b24d210773f62a58 P f90087404db048ccbc1bb901612f59d4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:51:31.170141 29924 raft_consensus.cc:2454] T 6976ffc957b64b31b24d210773f62a58 P 242d22b82213433a94d55a5104e65673 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 627f99588e0341d88d2272de309ddd6d in term 1.
I20260928 22:51:31.170220 30047 raft_consensus.cc:2454] T 6976ffc957b64b31b24d210773f62a58 P f90087404db048ccbc1bb901612f59d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 627f99588e0341d88d2272de309ddd6d in term 1.
I20260928 22:51:31.170324 29725 leader_election.cc:302] T 6976ffc957b64b31b24d210773f62a58 P 627f99588e0341d88d2272de309ddd6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 242d22b82213433a94d55a5104e65673, 627f99588e0341d88d2272de309ddd6d; no voters:
I20260928 22:51:31.170478 30143 raft_consensus.cc:2781] T 6976ffc957b64b31b24d210773f62a58 P 627f99588e0341d88d2272de309ddd6d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:51:31.170631 30143 raft_consensus.cc:686] T 6976ffc957b64b31b24d210773f62a58 P 627f99588e0341d88d2272de309ddd6d [term 1 LEADER]: Becoming Leader. State: Replica: 627f99588e0341d88d2272de309ddd6d, State: Running, Role: LEADER
I20260928 22:51:31.170724 30143 consensus_queue.cc:237] T 6976ffc957b64b31b24d210773f62a58 P 627f99588e0341d88d2272de309ddd6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f90087404db048ccbc1bb901612f59d4" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 46089 } } peers { permanent_uuid: "627f99588e0341d88d2272de309ddd6d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 39337 } } peers { permanent_uuid: "242d22b82213433a94d55a5104e65673" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 43063 } }
I20260928 22:51:31.171530 29638 catalog_manager.cc:5495] T 6976ffc957b64b31b24d210773f62a58 P 627f99588e0341d88d2272de309ddd6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 627f99588e0341d88d2272de309ddd6d (127.16.121.129). New cstate: current_term: 1 leader_uuid: "627f99588e0341d88d2272de309ddd6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f90087404db048ccbc1bb901612f59d4" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 46089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "627f99588e0341d88d2272de309ddd6d" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 39337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "242d22b82213433a94d55a5104e65673" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 43063 } health_report { overall_health: UNKNOWN } } }
W20260928 22:51:31.175171 30105 tablet.cc:2369] T 6976ffc957b64b31b24d210773f62a58 P f90087404db048ccbc1bb901612f59d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:51:31.211395 30135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:31.208056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54238 (local address 127.16.121.129:39337)
0928 22:51:31.208125 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:31.208128 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:31.208158 (+ 30us) server_negotiation.cc:407] Connection header received
0928 22:51:31.208203 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:31.208204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:31.208226 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:31.208300 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:51:31.208476 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:31.209264 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:31.210364 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:31.210668 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:31.210735 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:31.210941 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:51:31.211118 (+ 177us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:51:31.211218 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:31.211270 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 22:51:31.211296 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 22:51:31.214057 29924 raft_consensus.cc:1261] T 6976ffc957b64b31b24d210773f62a58 P 242d22b82213433a94d55a5104e65673 [term 1 FOLLOWER]: Refusing update from remote peer 627f99588e0341d88d2272de309ddd6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:51:31.214474 30143 consensus_queue.cc:1035] T 6976ffc957b64b31b24d210773f62a58 P 627f99588e0341d88d2272de309ddd6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "242d22b82213433a94d55a5104e65673" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 43063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:51:31.217060 30047 raft_consensus.cc:1261] T 6976ffc957b64b31b24d210773f62a58 P f90087404db048ccbc1bb901612f59d4 [term 1 FOLLOWER]: Refusing update from remote peer 627f99588e0341d88d2272de309ddd6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:51:31.217487 30143 consensus_queue.cc:1035] T 6976ffc957b64b31b24d210773f62a58 P 627f99588e0341d88d2272de309ddd6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f90087404db048ccbc1bb901612f59d4" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 46089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:51:31.219180 30151 mvcc.cc:205] Tried to move back new op lower bound from 7334444610409594880 to 7334444610235764736. Current Snapshot: MvccSnapshot[applied={T|T < 7334444610409594880}]
I20260928 22:51:31.221959 30154 mvcc.cc:205] Tried to move back new op lower bound from 7334444610409594880 to 7334444610235764736. Current Snapshot: MvccSnapshot[applied={T|T < 7334444610409594880}]
I20260928 22:51:31.222508 30152 mvcc.cc:205] Tried to move back new op lower bound from 7334444610409594880 to 7334444610235764736. Current Snapshot: MvccSnapshot[applied={T|T < 7334444610409594880}]
I20260928 22:51:31.225363 29638 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:51:31.225473 29638 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:51:31.226233 29638 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a50a70da076c47daa85d718844995547: Sending DeleteTablet for 3 replicas of tablet 6976ffc957b64b31b24d210773f62a58
I20260928 22:51:31.226668 30038 tablet_service.cc:1514] Processing DeleteTablet for tablet 6976ffc957b64b31b24d210773f62a58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:51:31 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:60962
I20260928 22:51:31.227123 29904 tablet_service.cc:1514] Processing DeleteTablet for tablet 6976ffc957b64b31b24d210773f62a58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:51:31 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:49592
I20260928 22:51:31.227481 29770 tablet_service.cc:1514] Processing DeleteTablet for tablet 6976ffc957b64b31b24d210773f62a58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:51:31 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:54224
I20260928 22:51:31.227944 30161 tablet_replica.cc:330] T 6976ffc957b64b31b24d210773f62a58 P 242d22b82213433a94d55a5104e65673: stopping tablet replica
I20260928 22:51:31.228067 30161 raft_consensus.cc:2229] T 6976ffc957b64b31b24d210773f62a58 P 242d22b82213433a94d55a5104e65673 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:51:31.228087 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 29676
I20260928 22:51:31.228201 30161 raft_consensus.cc:2258] T 6976ffc957b64b31b24d210773f62a58 P 242d22b82213433a94d55a5104e65673 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:51:31.229224 30161 ts_tablet_manager.cc:1905] T 6976ffc957b64b31b24d210773f62a58 P 242d22b82213433a94d55a5104e65673: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:51:31.231447 30161 ts_tablet_manager.cc:1918] T 6976ffc957b64b31b24d210773f62a58 P 242d22b82213433a94d55a5104e65673: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:51:31.231523 30161 log.cc:1196] T 6976ffc957b64b31b24d210773f62a58 P 242d22b82213433a94d55a5104e65673: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-1/wal/wals/6976ffc957b64b31b24d210773f62a58
I20260928 22:51:31.231808 30161 ts_tablet_manager.cc:1939] T 6976ffc957b64b31b24d210773f62a58 P 242d22b82213433a94d55a5104e65673: Deleting consensus metadata
I20260928 22:51:31.232028 30160 tablet_replica.cc:330] T 6976ffc957b64b31b24d210773f62a58 P f90087404db048ccbc1bb901612f59d4: stopping tablet replica
I20260928 22:51:31.232156 30160 raft_consensus.cc:2229] T 6976ffc957b64b31b24d210773f62a58 P f90087404db048ccbc1bb901612f59d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:51:31.232307 30160 raft_consensus.cc:2258] T 6976ffc957b64b31b24d210773f62a58 P f90087404db048ccbc1bb901612f59d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:51:31.232815 30160 ts_tablet_manager.cc:1905] T 6976ffc957b64b31b24d210773f62a58 P f90087404db048ccbc1bb901612f59d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:51:31.234447 30160 ts_tablet_manager.cc:1918] T 6976ffc957b64b31b24d210773f62a58 P f90087404db048ccbc1bb901612f59d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:51:31.234535 30160 log.cc:1196] T 6976ffc957b64b31b24d210773f62a58 P f90087404db048ccbc1bb901612f59d4: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790635733329745-16870-0/minicluster-data/ts-2/wal/wals/6976ffc957b64b31b24d210773f62a58
I20260928 22:51:31.234833 30160 ts_tablet_manager.cc:1939] T 6976ffc957b64b31b24d210773f62a58 P f90087404db048ccbc1bb901612f59d4: Deleting consensus metadata
I20260928 22:51:31.235417 29624 catalog_manager.cc:4841] TS 242d22b82213433a94d55a5104e65673 (127.16.121.130:43063): tablet 6976ffc957b64b31b24d210773f62a58 (table test-table [id=ce81de63c0a34c52983509afb94b2eb6]) successfully deleted
I20260928 22:51:31.236024 29625 catalog_manager.cc:4841] TS f90087404db048ccbc1bb901612f59d4 (127.16.121.131:46089): tablet 6976ffc957b64b31b24d210773f62a58 (table test-table [id=ce81de63c0a34c52983509afb94b2eb6]) successfully deleted
W20260928 22:51:31.239876 29624 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.121.129:39337 (error 108)
I20260928 22:51:31.240514 30131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:31.240395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:39337 (local address 127.0.0.1:54254)
0928 22:51:31.240436 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:31.240470 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.129:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260928 22:51:31.240667 29624 catalog_manager.cc:4584] TS 627f99588e0341d88d2272de309ddd6d (127.16.121.129:39337): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6976ffc957b64b31b24d210773f62a58: Network error: Client connection negotiation failed: client connection to 127.16.121.129:39337: connect: Connection refused (error 111)
I20260928 22:51:31.240986 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 29840
I20260928 22:51:31.247522 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 29973
I20260928 22:51:31.252329 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 29605
I20260928 22:51:31.258076 16870 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 22:51:31.603338 16870 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 22:51:31.603430 16870 mini_ranger.cc:66] Stopping Ranger...
I20260928 22:51:31.948340 16870 mini_ranger.cc:68] Stopped Ranger
2026-09-28 22:51:31.948 UTC [28672] LOG: received smart shutdown request
2026-09-28 22:51:31.952 UTC [28672] LOG: background worker "logical replication launcher" (PID 28681) exited with exit code 1
2026-09-28 22:51:31.953 UTC [28676] LOG: shutting down
2026-09-28 22:51:32.164 UTC [28672] LOG: database system is shut down
2026-09-28T22:51:32Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51586 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:51:32 dist-test-slave-56vw krb5kdc[30173](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:51:32 dist-test-slave-56vw krb5kdc[30173](info): set up 2 sockets
Sep 28 22:51:32 dist-test-slave-56vw krb5kdc[30173](info): commencing operation
krb5kdc: starting...
W20260928 22:51:33.331282 16870 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Sep 28 22:51:33 dist-test-slave-56vw krb5kdc[30173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:51:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:51:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.121.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
I20260928 22:51:33.459376 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:45555
--webserver_interface=127.16.121.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:37463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:45555,127.16.121.189:36583,127.16.121.188:45567
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:51:33.543426 30186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:51:33.543676 30186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:51:33.543716 30186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:51:33.544920 30186 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:51:33.544993 30186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:51:33.545017 30186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:51:33.545047 30186 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:51:33.545076 30186 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:51:33.546641 30186 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:37463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:45555,127.16.121.189:36583,127.16.121.188:45567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:45555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.121.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:51:33.546936 30186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:51:33.547202 30186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:51:33.552371 30192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:51:33.552377 30194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:51:33.552688 30186 server_base.cc:1034] running on GCE node
W20260928 22:51:33.552417 30191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:51:33.553079 30186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:51:33.553406 30186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:51:33.554590 30186 hybrid_clock.cc:648] HybridClock initialized: now 1790635893554560 us; error 59 us; skew 500 ppm
Sep 28 22:51:33 dist-test-slave-56vw krb5kdc[30173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635893, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:51:33.557363 30186 init.cc:377] Logged in from keytab as kudu/127.16.121.190@KRBTEST.COM (short username kudu)
I20260928 22:51:33.557814 30186 webserver.cc:460] Webserver started at http://127.16.121.190:41401/ using document root <none> and password file <none>
I20260928 22:51:33.558071 30186 fs_manager.cc:359] Metadata directory not provided
I20260928 22:51:33.558141 30186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:51:33.558295 30186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:51:33.559491 30186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "51ce6a9e63ff44bf8e3947425a77cd7f"
format_stamp: "Formatted at 2026-09-28 22:51:33 on dist-test-slave-56vw"
server_key: "c72e97224ad145ff7178cc2cfab599c2"
server_key_iv: "7181fe791175fe9146b1a355d4e52ab9"
server_key_version: "encryptionkey@0"
I20260928 22:51:33.559852 30186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "51ce6a9e63ff44bf8e3947425a77cd7f"
format_stamp: "Formatted at 2026-09-28 22:51:33 on dist-test-slave-56vw"
server_key: "c72e97224ad145ff7178cc2cfab599c2"
server_key_iv: "7181fe791175fe9146b1a355d4e52ab9"
server_key_version: "encryptionkey@0"
I20260928 22:51:33.561268 30186 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:51:33.562107 30201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:33.562311 30186 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:51:33.562417 30186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "51ce6a9e63ff44bf8e3947425a77cd7f"
format_stamp: "Formatted at 2026-09-28 22:51:33 on dist-test-slave-56vw"
server_key: "c72e97224ad145ff7178cc2cfab599c2"
server_key_iv: "7181fe791175fe9146b1a355d4e52ab9"
server_key_version: "encryptionkey@0"
I20260928 22:51:33.562492 30186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:51:33.572589 30186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:51:33.574661 30186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:51:33.574817 30186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:51:33.578660 30186 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.190:45555
I20260928 22:51:33.578732 30253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.190:45555 every 8 connection(s)
I20260928 22:51:33.579164 30186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:51:33.580127 30254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:51:33.581758 30254 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.121.190" port: 45555 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:51:33.584688 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 30186
I20260928 22:51:33.584846 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:51:33.585044 16870 external_mini_cluster.cc:1428] Setting key ed04bd0860fb6fd55b52e606d09fb3e8
Sep 28 22:51:33 dist-test-slave-56vw krb5kdc[30173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635893, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:51:33.594198 30256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:33.582535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:45555 (local address 127.0.0.1:52722)
0928 22:51:33.582700 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:33.582741 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 22:51:33.583326 (+ 585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:51:33.583597 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:51:33.583600 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:51:33.583873 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:51:33.584035 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:33.584039 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:33.585278 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:33.585280 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:33.586483 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:33.586486 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:33.587310 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:33.587311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:33.587379 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:33.588014 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:51:33.588025 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:51:33.589772 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:51:33.592178 (+ 2406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:33.592185 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:33.592187 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:33.592437 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:33.592755 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:33.592755 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:33.592756 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:33.592845 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:33.593434 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:51:33.593442 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:51:33.593515 (+ 73us) client_negotiation.cc:769] Sending connection context
0928 22:51:33.593846 (+ 331us) client_negotiation.cc:240] Negotiation successful
0928 22:51:33.594009 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":68,"threads_started":1}
I20260928 22:51:33.594287 30257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:33.582701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52722 (local address 127.16.121.190:45555)
0928 22:51:33.582816 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:33.582820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:33.582861 (+ 41us) server_negotiation.cc:407] Connection header received
0928 22:51:33.583470 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:33.583472 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:33.583489 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:33.583570 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:51:33.584152 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:33.585166 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:33.586613 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:33.587199 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:33.587345 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:33.589878 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:51:33.589888 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:51:33.589889 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:51:33.589897 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:51:33.592064 (+ 2167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:33.592540 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:33.592544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:33.592544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:33.592674 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:33.592922 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:33.592923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:33.592923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:33.593327 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:51:33.593428 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:33.593920 (+ 492us) server_negotiation.cc:299] Negotiation successful
0928 22:51:33.594107 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":86,"thread_start_us":60,"threads_started":1}
WARNING: no policy specified for kudu/127.16.121.189@KRBTEST.COM; defaulting to no policy
I20260928 22:51:33.595254 30254 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.121.189" port: 36583 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:51:33.595706 30256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:33.595564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.189:36583 (local address 127.0.0.1:55322)
0928 22:51:33.595624 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:33.595685 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.189:36583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 22:51:33.595894 30204 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.121.189:36583: connect: Connection refused (error 111)
I20260928 22:51:33.596402 30256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:33.596312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.189:36583 (local address 127.0.0.1:55328)
0928 22:51:33.596361 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:33.596387 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.189:36583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 22:51:33.596565 30254 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.121.189:36583: Network error: Client connection negotiation failed: client connection to 127.16.121.189:36583: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.189.keytab.
Entry for principal kudu/127.16.121.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.189.keytab.
I20260928 22:51:33.615206 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.189:36583
--webserver_interface=127.16.121.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.189.keytab
--principal=kudu/127.16.121.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:37463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:45555,127.16.121.189:36583,127.16.121.188:45567
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:51:33.645824 30254 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.121.189" port: 36583 } attempt: 1
I20260928 22:51:33.646375 30256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:33.646204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.189:36583 (local address 127.0.0.1:55330)
0928 22:51:33.646279 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:33.646333 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.189:36583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 22:51:33.646787 30256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:33.646684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.189:36583 (local address 127.0.0.1:55340)
0928 22:51:33.646735 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:33.646766 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.189:36583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 22:51:33.646938 30254 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.121.189:36583: Network error: Client connection negotiation failed: client connection to 127.16.121.189:36583: connect: Connection refused (error 111)
W20260928 22:51:33.698005 30261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:51:33.698230 30261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:51:33.698254 30261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:51:33.699517 30261 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:51:33.699563 30261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:51:33.699579 30261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:51:33.699591 30261 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:51:33.699604 30261 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:51:33.701108 30261 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:37463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:45555,127.16.121.189:36583,127.16.121.188:45567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.189.keytab
--principal=kudu/127.16.121.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.189:36583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.121.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:51:33.701376 30261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:51:33.701614 30261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:51:33.706744 30266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:51:33.706810 30269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:51:33.706801 30261 server_base.cc:1034] running on GCE node
W20260928 22:51:33.706765 30267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:51:33.707280 30261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:51:33.707594 30261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:51:33.708761 30261 hybrid_clock.cc:648] HybridClock initialized: now 1790635893708740 us; error 45 us; skew 500 ppm
Sep 28 22:51:33 dist-test-slave-56vw krb5kdc[30173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635893, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:51:33.711663 30261 init.cc:377] Logged in from keytab as kudu/127.16.121.189@KRBTEST.COM (short username kudu)
I20260928 22:51:33.712167 30261 webserver.cc:460] Webserver started at http://127.16.121.189:45855/ using document root <none> and password file <none>
I20260928 22:51:33.712402 30261 fs_manager.cc:359] Metadata directory not provided
I20260928 22:51:33.712472 30261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:51:33.712627 30261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:51:33.713757 30261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-1/data/instance:
uuid: "f829a3a5cb534641a40e19edef77478d"
format_stamp: "Formatted at 2026-09-28 22:51:33 on dist-test-slave-56vw"
server_key: "e36842eedc0a026813249a6771b01f12"
server_key_iv: "a761b5b683ec907b21cd8fd346b92f49"
server_key_version: "encryptionkey@0"
I20260928 22:51:33.714129 30261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-1/wal/instance:
uuid: "f829a3a5cb534641a40e19edef77478d"
format_stamp: "Formatted at 2026-09-28 22:51:33 on dist-test-slave-56vw"
server_key: "e36842eedc0a026813249a6771b01f12"
server_key_iv: "a761b5b683ec907b21cd8fd346b92f49"
server_key_version: "encryptionkey@0"
I20260928 22:51:33.715113 30254 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.121.189" port: 36583 } attempt: 2
I20260928 22:51:33.715541 30256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:33.715377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.189:36583 (local address 127.0.0.1:55346)
0928 22:51:33.715451 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:33.715486 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.189:36583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 22:51:33.715610 30261 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:51:33.715953 30256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:33.715827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.189:36583 (local address 127.0.0.1:55358)
0928 22:51:33.715902 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:33.715933 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.189:36583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 22:51:33.716130 30254 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.121.189:36583: Network error: Client connection negotiation failed: client connection to 127.16.121.189:36583: connect: Connection refused (error 111)
I20260928 22:51:33.716476 30276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:33.716897 30261 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:51:33.717034 30261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-1/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-1/wal
uuid: "f829a3a5cb534641a40e19edef77478d"
format_stamp: "Formatted at 2026-09-28 22:51:33 on dist-test-slave-56vw"
server_key: "e36842eedc0a026813249a6771b01f12"
server_key_iv: "a761b5b683ec907b21cd8fd346b92f49"
server_key_version: "encryptionkey@0"
I20260928 22:51:33.717133 30261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:51:33.726245 30261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:51:33.728776 30261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:51:33.728969 30261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:51:33.733188 30261 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.189:36583
I20260928 22:51:33.733232 30328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.189:36583 every 8 connection(s)
I20260928 22:51:33.733603 30261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-1/data/info.pb
I20260928 22:51:33.734516 30329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:51:33.736220 30329 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.121.190" port: 45555 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:51:33.740389 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 30261
I20260928 22:51:33.740507 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-1/wal/instance
I20260928 22:51:33.740715 16870 external_mini_cluster.cc:1428] Setting key c94268c4f6202842390eb04d5b9a3538
Sep 28 22:51:33 dist-test-slave-56vw krb5kdc[30173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635893, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.189@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:51:33.747224 30257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:33.737064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52736 (local address 127.16.121.190:45555)
0928 22:51:33.737154 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:33.737156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:33.737374 (+ 218us) server_negotiation.cc:407] Connection header received
0928 22:51:33.737884 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:33.737886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:33.737901 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:33.737965 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:51:33.738602 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:33.739296 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:33.740533 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:33.740761 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:33.740822 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:33.743743 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:51:33.743744 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:51:33.743745 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:51:33.743754 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:51:33.745446 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:33.746011 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:33.746012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:33.746013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:33.746072 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:33.746397 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:33.746398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:33.746398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:33.746514 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:51:33.746653 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:33.746977 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 22:51:33.747127 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 22:51:33.747417 30331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:33.737018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:45555 (local address 127.0.0.1:52736)
0928 22:51:33.737219 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:33.737276 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 22:51:33.737774 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:51:33.738042 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:51:33.738045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:51:33.738305 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:51:33.738498 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:33.738503 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:33.739445 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:33.739447 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:33.740431 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:33.740435 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:33.740859 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:33.740860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:33.741021 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:33.741591 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:51:33.741609 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:51:33.743628 (+ 2019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:51:33.745540 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:33.745549 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:33.745552 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:33.745919 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:33.746162 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:33.746163 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:33.746163 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:33.746324 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:33.746598 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:51:33.746606 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:51:33.746798 (+ 192us) client_negotiation.cc:769] Sending connection context
0928 22:51:33.746928 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 22:51:33.747144 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":89,"threads_started":1}
I20260928 22:51:33.748183 30329 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.121.189" port: 36583 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.121.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.188@KRBTEST.COM" created.
Sep 28 22:51:33 dist-test-slave-56vw krb5kdc[30173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635893, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.189@KRBTEST.COM for kudu/127.16.121.189@KRBTEST.COM
I20260928 22:51:33.757894 30331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:33.748490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.189:36583 (local address 127.0.0.1:55370)
0928 22:51:33.748550 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:33.748564 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 22:51:33.748612 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:51:33.748987 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:51:33.748988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:51:33.749145 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:51:33.749194 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:33.749195 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:33.750261 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:33.750262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:33.750847 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:33.750849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:33.751463 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:33.751463 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:33.751525 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:33.753160 (+ 1635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:51:33.753164 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:51:33.754379 (+ 1215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:51:33.756702 (+ 2323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:33.756703 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:33.756703 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:33.756865 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:33.757125 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:33.757126 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:33.757127 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:33.757188 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:33.757556 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:51:33.757557 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:51:33.757668 (+ 111us) client_negotiation.cc:769] Sending connection context
0928 22:51:33.757734 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 22:51:33.757817 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 22:51:33.758042 30333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:33.748658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55370 (local address 127.16.121.189:36583)
0928 22:51:33.748824 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:33.748829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:33.748847 (+ 18us) server_negotiation.cc:407] Connection header received
0928 22:51:33.748875 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:33.748877 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:33.748909 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:33.748988 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:51:33.749272 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:33.750155 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:33.750962 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:33.751339 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:33.751479 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:33.754477 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:51:33.754485 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:51:33.754485 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:51:33.754492 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:51:33.756576 (+ 2084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:33.756952 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:33.756957 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:33.756958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:33.757029 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:33.757272 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:33.757272 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:33.757273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:33.757478 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:51:33.757574 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:33.757800 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 22:51:33.757894 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":108,"threads_started":1}
I20260928 22:51:33.760264 30329 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.121.188" port: 45567 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:51:33.760668 30331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:33.760530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:45567 (local address 127.0.0.1:37634)
0928 22:51:33.760583 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:33.760643 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.188:45567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 22:51:33.760807 30278 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.121.188:45567: connect: Connection refused (error 111)
I20260928 22:51:33.761305 30331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:33.761207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:45567 (local address 127.0.0.1:37636)
0928 22:51:33.761258 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:33.761282 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.188:45567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 22:51:33.761425 30329 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.121.188:45567: Network error: Client connection negotiation failed: client connection to 127.16.121.188:45567: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.188.keytab.
Entry for principal kudu/127.16.121.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.188.keytab.
I20260928 22:51:33.770967 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.188:45567
--webserver_interface=127.16.121.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.188.keytab
--principal=kudu/127.16.121.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:37463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:45555,127.16.121.189:36583,127.16.121.188:45567
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:51:33.807308 30254 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.121.189" port: 36583 } attempt: 3
I20260928 22:51:33.810667 30329 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.121.188" port: 45567 } attempt: 1
I20260928 22:51:33.811534 30331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:33.811384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:45567 (local address 127.0.0.1:37642)
0928 22:51:33.811451 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:33.811492 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.188:45567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 22:51:33.811977 30331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:33.811857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:45567 (local address 127.0.0.1:37654)
0928 22:51:33.811916 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:33.811948 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.188:45567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
Sep 28 22:51:33 dist-test-slave-56vw krb5kdc[30173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635893, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for kudu/127.16.121.189@KRBTEST.COM
W20260928 22:51:33.812705 30329 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.121.188:45567: Network error: Client connection negotiation failed: client connection to 127.16.121.188:45567: connect: Connection refused (error 111)
I20260928 22:51:33.815840 30256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:33.807788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.189:36583 (local address 127.0.0.1:55376)
0928 22:51:33.807855 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:33.807868 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:51:33.807934 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:51:33.808148 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:51:33.808149 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:51:33.808307 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:51:33.808380 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:33.808382 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:33.809383 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:33.809384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:33.809996 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:33.809999 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:33.810517 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:33.810518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:33.810747 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:33.811336 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:51:33.811342 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:51:33.812941 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:51:33.814642 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:33.814643 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:33.814644 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:33.814817 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:33.815017 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:33.815018 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:33.815018 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:33.815116 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:33.815410 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:51:33.815411 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:51:33.815471 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 22:51:33.815662 (+ 191us) client_negotiation.cc:240] Negotiation successful
0928 22:51:33.815772 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 22:51:33.815877 30333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:33.807840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55376 (local address 127.16.121.189:36583)
0928 22:51:33.807902 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:33.807904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:33.807922 (+ 18us) server_negotiation.cc:407] Connection header received
0928 22:51:33.808018 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:33.808020 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:33.808036 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:33.808115 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:51:33.808494 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:33.809292 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:33.810089 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:33.810414 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:33.810489 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:33.813027 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:51:33.813028 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:51:33.813028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:51:33.813036 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:51:33.814551 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:33.814889 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:33.814890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:33.814890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:33.814951 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:33.815188 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:33.815189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:33.815189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:33.815321 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:51:33.815389 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:33.815703 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 22:51:33.815794 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 22:51:33.816363 30254 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.121.188" port: 45567 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:51:33.816753 30256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:33.816619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:45567 (local address 127.0.0.1:37656)
0928 22:51:33.816674 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:33.816712 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.188:45567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 22:51:33.817121 30256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:33.817022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:45567 (local address 127.0.0.1:37662)
0928 22:51:33.817074 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:33.817101 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.188:45567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 22:51:33.817256 30254 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.121.188:45567: Network error: Client connection negotiation failed: client connection to 127.16.121.188:45567: connect: Connection refused (error 111)
I20260928 22:51:33.848423 30254 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.121.188" port: 45567 } attempt: 1
I20260928 22:51:33.848994 30256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:33.848835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:45567 (local address 127.0.0.1:37676)
0928 22:51:33.848908 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:33.848956 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.188:45567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 22:51:33.849382 30256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:33.849285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:45567 (local address 127.0.0.1:37692)
0928 22:51:33.849331 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:33.849361 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.188:45567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 22:51:33.849480 30254 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.121.188:45567: Network error: Client connection negotiation failed: client connection to 127.16.121.188:45567: connect: Connection refused (error 111)
W20260928 22:51:33.857689 30336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:51:33.857903 30336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:51:33.857928 30336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:51:33.859104 30336 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:51:33.859156 30336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:51:33.859170 30336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:51:33.859203 30336 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:51:33.859215 30336 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:51:33.860703 30336 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:37463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:45555,127.16.121.189:36583,127.16.121.188:45567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.188.keytab
--principal=kudu/127.16.121.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.188:45567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.121.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:51:33.860909 30336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:51:33.861104 30336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:51:33.866259 30344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:51:33.866304 30341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:51:33.866415 30336 server_base.cc:1034] running on GCE node
W20260928 22:51:33.866443 30342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:51:33.866863 30336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:51:33.867223 30336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:51:33.868398 30336 hybrid_clock.cc:648] HybridClock initialized: now 1790635893868386 us; error 29 us; skew 500 ppm
Sep 28 22:51:33 dist-test-slave-56vw krb5kdc[30173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635893, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:51:33.871017 30336 init.cc:377] Logged in from keytab as kudu/127.16.121.188@KRBTEST.COM (short username kudu)
I20260928 22:51:33.871536 30336 webserver.cc:460] Webserver started at http://127.16.121.188:36809/ using document root <none> and password file <none>
I20260928 22:51:33.871773 30336 fs_manager.cc:359] Metadata directory not provided
I20260928 22:51:33.871843 30336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:51:33.872008 30336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:51:33.873179 30336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-2/data/instance:
uuid: "865f1ba635c94f54925b7fd3919fbcd0"
format_stamp: "Formatted at 2026-09-28 22:51:33 on dist-test-slave-56vw"
server_key: "45720aee1b368a74c4c074c87be9a576"
server_key_iv: "7a1f265b8b6901507d15720c05106c0b"
server_key_version: "encryptionkey@0"
I20260928 22:51:33.873553 30336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-2/wal/instance:
uuid: "865f1ba635c94f54925b7fd3919fbcd0"
format_stamp: "Formatted at 2026-09-28 22:51:33 on dist-test-slave-56vw"
server_key: "45720aee1b368a74c4c074c87be9a576"
server_key_iv: "7a1f265b8b6901507d15720c05106c0b"
server_key_version: "encryptionkey@0"
I20260928 22:51:33.874955 30336 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:51:33.875855 30351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:33.876075 30336 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:51:33.876209 30336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-2/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-2/wal
uuid: "865f1ba635c94f54925b7fd3919fbcd0"
format_stamp: "Formatted at 2026-09-28 22:51:33 on dist-test-slave-56vw"
server_key: "45720aee1b368a74c4c074c87be9a576"
server_key_iv: "7a1f265b8b6901507d15720c05106c0b"
server_key_version: "encryptionkey@0"
I20260928 22:51:33.876294 30336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:51:33.880853 30329 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.121.188" port: 45567 } attempt: 2
I20260928 22:51:33.881292 30331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:33.881140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:45567 (local address 127.0.0.1:37700)
0928 22:51:33.881211 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:33.881255 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.188:45567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 22:51:33.881711 30331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:33.881616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:45567 (local address 127.0.0.1:37712)
0928 22:51:33.881659 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:33.881691 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.188:45567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 22:51:33.881834 30329 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.121.188:45567: Network error: Client connection negotiation failed: client connection to 127.16.121.188:45567: connect: Connection refused (error 111)
I20260928 22:51:33.884856 30336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:51:33.887290 30336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:51:33.887451 30336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:51:33.891592 30336 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.188:45567
I20260928 22:51:33.891620 30403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.188:45567 every 8 connection(s)
I20260928 22:51:33.892000 30336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-2/data/info.pb
I20260928 22:51:33.892949 30404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:51:33.894507 30404 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.121.190" port: 45555 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:51:33.896938 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 30336
I20260928 22:51:33.897056 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790635733329745-16870-0/minicluster-data/master-2/wal/instance
I20260928 22:51:33.897243 16870 external_mini_cluster.cc:1428] Setting key 6f5820c4311ca05eeeea5ee251c38f5c
Sep 28 22:51:33 dist-test-slave-56vw krb5kdc[30173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635893, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.188@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
Sep 28 22:51:33 dist-test-slave-56vw krb5kdc[30173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.121.188@KRBTEST.COM
I20260928 22:51:33.905382 30257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:33.895312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52746 (local address 127.16.121.190:45555)
0928 22:51:33.895378 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:33.895380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:33.895548 (+ 168us) server_negotiation.cc:407] Connection header received
0928 22:51:33.895988 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:33.895990 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:33.895999 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:33.896060 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:51:33.896740 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:33.897325 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:33.898787 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:33.899105 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:33.899176 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:33.901805 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:51:33.901807 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:51:33.901807 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:51:33.901814 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:51:33.903624 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:33.904229 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:33.904231 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:33.904231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:33.904295 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:33.904580 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:33.904580 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:33.904581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:33.904730 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:51:33.904798 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:33.905154 (+ 356us) server_negotiation.cc:299] Negotiation successful
0928 22:51:33.905263 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 22:51:33.905562 30406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:33.895260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:45555 (local address 127.0.0.1:52746)
0928 22:51:33.895414 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:33.895456 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 22:51:33.895899 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:51:33.896111 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:51:33.896115 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:51:33.896366 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:51:33.896582 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:33.896586 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:33.897414 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:33.897415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:33.898684 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:33.898686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:33.899194 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:33.899195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:33.899307 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:33.899849 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:51:33.899859 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:51:33.901695 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:51:33.903741 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:33.903747 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:33.903749 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:33.904135 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:33.904365 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:33.904366 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:33.904366 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:33.904501 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:33.904798 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:51:33.904803 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:51:33.904975 (+ 172us) client_negotiation.cc:769] Sending connection context
0928 22:51:33.905104 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 22:51:33.905299 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":82,"threads_started":1}
I20260928 22:51:33.906320 30404 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.121.189" port: 36583 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:51:33.907681 30408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:33.897709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37728 (local address 127.16.121.188:45567)
0928 22:51:33.897850 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:33.897855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:33.897888 (+ 33us) server_negotiation.cc:407] Connection header received
0928 22:51:33.897959 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:33.897961 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:33.897977 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:33.898039 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:51:33.898780 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:33.899786 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:33.900824 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:33.901308 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:33.901377 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:33.904285 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:51:33.904291 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:51:33.904291 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:51:33.904297 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:51:33.906263 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:33.906678 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:33.906681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:33.906681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:33.906755 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:33.907090 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:33.907092 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:33.907092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:33.907287 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:51:33.907443 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:33.907497 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 22:51:33.907574 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":86,"threads_started":1}
Sep 28 22:51:33 dist-test-slave-56vw krb5kdc[30173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635893, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.188@KRBTEST.COM for kudu/127.16.121.189@KRBTEST.COM
Sep 28 22:51:33 dist-test-slave-56vw krb5kdc[30173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:51:33.913942 30406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:33.906618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.189:36583 (local address 127.0.0.1:55392)
0928 22:51:33.906712 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:33.906721 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:51:33.906757 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:51:33.906909 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:51:33.906909 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:51:33.907093 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:51:33.907145 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:33.907146 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:33.908031 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:33.908031 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:33.908636 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:33.908638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:33.909037 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:33.909038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:33.909097 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:33.909526 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:51:33.909531 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:51:33.910824 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:51:33.912795 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:33.912796 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:33.912796 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:33.913002 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:33.913210 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:33.913210 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:33.913211 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:33.913288 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:33.913582 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:51:33.913583 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:51:33.913633 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 22:51:33.913725 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 22:51:33.913816 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 22:51:33.914182 30333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:33.906673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55392 (local address 127.16.121.189:36583)
0928 22:51:33.906729 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:33.906731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:33.906744 (+ 13us) server_negotiation.cc:407] Connection header received
0928 22:51:33.906832 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:33.906833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:33.906844 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:33.906905 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:51:33.907228 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:33.907945 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:33.908742 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:33.908954 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:33.909040 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:33.910904 (+ 1864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:51:33.910906 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:51:33.910906 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:51:33.910914 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:51:33.912709 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:33.913075 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:33.913076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:33.913076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:33.913144 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:33.913358 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:33.913359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:33.913359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:33.913507 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:51:33.913572 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:33.913949 (+ 377us) server_negotiation.cc:299] Negotiation successful
0928 22:51:33.914059 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 22:51:33.914618 30404 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.121.188" port: 45567 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:51:33.916625 30257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:33.908603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52750 (local address 127.16.121.190:45555)
0928 22:51:33.908662 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:33.908664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:33.908672 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:51:33.908694 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:33.908695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:33.908703 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:33.908765 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:51:33.909214 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:33.909659 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:33.910627 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:33.910847 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:33.910949 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:33.913493 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:51:33.913494 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:51:33.913495 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:51:33.913501 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:51:33.915215 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:33.915571 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:33.915571 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:33.915572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:33.915621 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:33.915989 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:33.915991 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:33.915991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:33.916117 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:51:33.916379 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:33.916424 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 22:51:33.916523 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 22:51:33 dist-test-slave-56vw krb5kdc[30173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635893, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.188@KRBTEST.COM for kudu/127.16.121.188@KRBTEST.COM
Sep 28 22:51:33 dist-test-slave-56vw krb5kdc[30173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.121.189@KRBTEST.COM
I20260928 22:51:33.922897 30406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:33.914903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:45567 (local address 127.0.0.1:37732)
0928 22:51:33.914945 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:33.914956 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:51:33.915001 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:51:33.915180 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:51:33.915181 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:51:33.915330 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:51:33.915373 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:33.915374 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:33.916188 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:33.916189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:33.917119 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:33.917122 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:33.917627 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:33.917628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:33.917707 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:33.918065 (+ 358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:51:33.918071 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:51:33.919525 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:51:33.921693 (+ 2168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:33.921694 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:33.921694 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:33.921951 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:33.922165 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:33.922167 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:33.922167 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:33.922236 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:33.922537 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:51:33.922538 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:51:33.922578 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 22:51:33.922649 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 22:51:33.922754 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 22:51:33.922897 30408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:33.914928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37732 (local address 127.16.121.188:45567)
0928 22:51:33.914982 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:33.914984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:33.914993 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:51:33.915108 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:33.915109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:33.915120 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:33.915175 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:51:33.915444 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:33.916092 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:33.917236 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:33.917543 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:33.917622 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:33.919701 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:51:33.919702 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:51:33.919703 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:51:33.919711 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:51:33.921537 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:33.922037 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:33.922039 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:33.922039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:33.922092 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:33.922331 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:33.922332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:33.922332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:33.922442 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:51:33.922545 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:33.922678 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 22:51:33.922803 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":36}
I20260928 22:51:33.924513 30404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 865f1ba635c94f54925b7fd3919fbcd0: Bootstrap starting.
I20260928 22:51:33.924643 30254 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.121.188" port: 45567 } attempt: 2
I20260928 22:51:33.925547 30404 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 865f1ba635c94f54925b7fd3919fbcd0: Neither blocks nor log segments found. Creating new log.
I20260928 22:51:33.925854 30404 log.cc:824] T 00000000000000000000000000000000 P 865f1ba635c94f54925b7fd3919fbcd0: Log is configured to *not* fsync() on all Append() calls
I20260928 22:51:33.926821 30404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 865f1ba635c94f54925b7fd3919fbcd0: No bootstrap required, opened a new log
I20260928 22:51:33.928709 30333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:33.917474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55408 (local address 127.16.121.189:36583)
0928 22:51:33.917513 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:33.917515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:33.917523 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:51:33.917571 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:33.917572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:33.917582 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:33.917635 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:51:33.918322 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:33.918780 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:33.919715 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:33.919979 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:33.920047 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:33.922919 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:51:33.922921 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:51:33.922921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:51:33.922929 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:51:33.927417 (+ 4488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:33.927829 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:33.927830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:33.927831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:33.927906 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:33.928152 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:33.928153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:33.928153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:33.928293 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:51:33.928471 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:33.928509 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 22:51:33.928586 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:51:33.929165 30404 raft_consensus.cc:348] T 00000000000000000000000000000000 P 865f1ba635c94f54925b7fd3919fbcd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce6a9e63ff44bf8e3947425a77cd7f" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 45555 } } peers { permanent_uuid: "f829a3a5cb534641a40e19edef77478d" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 36583 } } peers { permanent_uuid: "865f1ba635c94f54925b7fd3919fbcd0" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 45567 } }
I20260928 22:51:33.929332 30404 raft_consensus.cc:374] T 00000000000000000000000000000000 P 865f1ba635c94f54925b7fd3919fbcd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:51:33.929390 30404 raft_consensus.cc:729] T 00000000000000000000000000000000 P 865f1ba635c94f54925b7fd3919fbcd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 865f1ba635c94f54925b7fd3919fbcd0, State: Initialized, Role: FOLLOWER
I20260928 22:51:33.929523 30404 consensus_queue.cc:260] T 00000000000000000000000000000000 P 865f1ba635c94f54925b7fd3919fbcd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce6a9e63ff44bf8e3947425a77cd7f" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 45555 } } peers { permanent_uuid: "f829a3a5cb534641a40e19edef77478d" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 36583 } } peers { permanent_uuid: "865f1ba635c94f54925b7fd3919fbcd0" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 45567 } }
I20260928 22:51:33.929811 30410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 865f1ba635c94f54925b7fd3919fbcd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce6a9e63ff44bf8e3947425a77cd7f" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 45555 } } peers { permanent_uuid: "f829a3a5cb534641a40e19edef77478d" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 36583 } } peers { permanent_uuid: "865f1ba635c94f54925b7fd3919fbcd0" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 45567 } } }
I20260928 22:51:33.929957 30410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 865f1ba635c94f54925b7fd3919fbcd0 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:51:33.929879 30404 sys_catalog.cc:564] T 00000000000000000000000000000000 P 865f1ba635c94f54925b7fd3919fbcd0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:51:33.930361 16870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
Sep 28 22:51:33 dist-test-slave-56vw krb5kdc[30173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635893, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for kudu/127.16.121.188@KRBTEST.COM
W20260928 22:51:33.934083 30421 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 865f1ba635c94f54925b7fd3919fbcd0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:51:33.934216 30421 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:51:33.934934 30256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:33.924952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:45567 (local address 127.0.0.1:37744)
0928 22:51:33.925009 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:33.925019 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:51:33.925059 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:51:33.926366 (+ 1307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:51:33.926368 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:51:33.926890 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:51:33.926947 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:33.926950 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:33.927898 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:33.927899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:33.928640 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:33.928643 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:33.929142 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:33.929143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:33.929202 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:33.929622 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:51:33.929627 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:51:33.931155 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:51:33.933734 (+ 2579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:33.933735 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:33.933736 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:33.933954 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:33.934182 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:33.934183 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:33.934184 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:33.934257 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:33.934590 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:51:33.934591 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:51:33.934661 (+ 70us) client_negotiation.cc:769] Sending connection context
0928 22:51:33.934712 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 22:51:33.934810 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":7}
I20260928 22:51:33.932406 30336 master_runner.cc:186] Error getting master registration for 127.16.121.190:45555: OK, instance_id { permanent_uuid: "51ce6a9e63ff44bf8e3947425a77cd7f" instance_seqno: 1790635893577562 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 22:51:33.935511 30336 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 22:51:33.935683 30408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:33.926135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37744 (local address 127.16.121.188:45567)
0928 22:51:33.926217 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:33.926219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:33.926229 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:51:33.926251 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:33.926252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:33.926264 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:33.926334 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:51:33.927037 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:33.927759 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:33.928734 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:33.929019 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:33.929139 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:33.931256 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:51:33.931258 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:51:33.931258 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:51:33.931268 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:51:33.933640 (+ 2372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:33.934043 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:33.934044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:33.934045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:33.934102 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:33.934333 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:33.934334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:33.934335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:33.934499 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:51:33.935410 (+ 911us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:33.935450 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 22:51:33.935562 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 22:51:33.937970 30254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51ce6a9e63ff44bf8e3947425a77cd7f: Bootstrap starting.
I20260928 22:51:33.938804 30254 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51ce6a9e63ff44bf8e3947425a77cd7f: Neither blocks nor log segments found. Creating new log.
I20260928 22:51:33.939210 30254 log.cc:824] T 00000000000000000000000000000000 P 51ce6a9e63ff44bf8e3947425a77cd7f: Log is configured to *not* fsync() on all Append() calls
I20260928 22:51:33.940924 30333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:33.932455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55412 (local address 127.16.121.189:36583)
0928 22:51:33.932496 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:33.932498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:33.932687 (+ 189us) server_negotiation.cc:407] Connection header received
0928 22:51:33.932721 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:33.932722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:33.932733 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:33.932789 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:51:33.933495 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:33.934231 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:33.935277 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:33.935554 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:33.935602 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:33.937332 (+ 1730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:51:33.937333 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:51:33.937334 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:51:33.937342 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:51:33.939040 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:33.939556 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:33.939558 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:33.939558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:33.939636 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:33.939908 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:33.939909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:33.939910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:33.940036 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:51:33.940684 (+ 648us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:33.940726 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 22:51:33.940804 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:51:33.940950 30254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51ce6a9e63ff44bf8e3947425a77cd7f: No bootstrap required, opened a new log
I20260928 22:51:33.943354 30254 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51ce6a9e63ff44bf8e3947425a77cd7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce6a9e63ff44bf8e3947425a77cd7f" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 45555 } } peers { permanent_uuid: "f829a3a5cb534641a40e19edef77478d" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 36583 } } peers { permanent_uuid: "865f1ba635c94f54925b7fd3919fbcd0" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 45567 } }
I20260928 22:51:33.943568 30254 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51ce6a9e63ff44bf8e3947425a77cd7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:51:33.943657 30254 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51ce6a9e63ff44bf8e3947425a77cd7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51ce6a9e63ff44bf8e3947425a77cd7f, State: Initialized, Role: FOLLOWER
I20260928 22:51:33.943730 30428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:33.933272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37748 (local address 127.16.121.188:45567)
0928 22:51:33.934362 (+ 1090us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:33.934364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:33.936162 (+ 1798us) server_negotiation.cc:407] Connection header received
0928 22:51:33.936183 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:33.936183 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:33.936195 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:33.936254 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:51:33.937038 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:33.937916 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:33.938922 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:33.939287 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:33.939352 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:33.940913 (+ 1561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:51:33.940914 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:51:33.940914 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:51:33.940921 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:51:33.942449 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:33.942817 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:33.942818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:33.942818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:33.942873 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:33.943189 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:33.943190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:33.943191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:33.943304 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:51:33.943410 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:33.943528 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 22:51:33.943612 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1065,"thread_start_us":75,"threads_started":1}
I20260928 22:51:33.943873 30254 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51ce6a9e63ff44bf8e3947425a77cd7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce6a9e63ff44bf8e3947425a77cd7f" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 45555 } } peers { permanent_uuid: "f829a3a5cb534641a40e19edef77478d" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 36583 } } peers { permanent_uuid: "865f1ba635c94f54925b7fd3919fbcd0" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 45567 } }
I20260928 22:51:33.944244 30432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51ce6a9e63ff44bf8e3947425a77cd7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce6a9e63ff44bf8e3947425a77cd7f" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 45555 } } peers { permanent_uuid: "f829a3a5cb534641a40e19edef77478d" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 36583 } } peers { permanent_uuid: "865f1ba635c94f54925b7fd3919fbcd0" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 45567 } } }
I20260928 22:51:33.944347 30432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51ce6a9e63ff44bf8e3947425a77cd7f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:51:33.944404 30257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:33.933024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52760 (local address 127.16.121.190:45555)
0928 22:51:33.933068 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:33.933070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:33.934478 (+ 1408us) server_negotiation.cc:407] Connection header received
0928 22:51:33.934576 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:33.934577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:33.934588 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:33.934618 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:51:33.935415 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:33.936390 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:33.937465 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:33.938030 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:33.938069 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:33.940609 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:51:33.940611 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:51:33.940611 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:51:33.940618 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:51:33.942711 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:33.943151 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:33.943152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:33.943153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:33.943207 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:33.943413 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:33.943414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:33.943415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:33.943539 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:51:33.943615 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:33.944123 (+ 508us) server_negotiation.cc:299] Negotiation successful
0928 22:51:33.944209 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":23}
I20260928 22:51:33.944308 30254 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51ce6a9e63ff44bf8e3947425a77cd7f [sys.catalog]: configured and running, proceeding with master startup.
W20260928 22:51:33.946296 30443 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 51ce6a9e63ff44bf8e3947425a77cd7f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:51:33.946398 30443 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:51:33.947022 30186 master_runner.cc:186] Error getting master registration for 127.16.121.189:36583: OK, instance_id { permanent_uuid: "f829a3a5cb534641a40e19edef77478d" instance_seqno: 1790635893732010 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 22:51:33.949313 30186 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 22:51:33.973001 30329 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.121.188" port: 45567 } attempt: 3
I20260928 22:51:33.973275 30410 raft_consensus.cc:482] T 00000000000000000000000000000000 P 865f1ba635c94f54925b7fd3919fbcd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:51:33.973395 30410 raft_consensus.cc:504] T 00000000000000000000000000000000 P 865f1ba635c94f54925b7fd3919fbcd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce6a9e63ff44bf8e3947425a77cd7f" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 45555 } } peers { permanent_uuid: "f829a3a5cb534641a40e19edef77478d" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 36583 } } peers { permanent_uuid: "865f1ba635c94f54925b7fd3919fbcd0" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 45567 } }
I20260928 22:51:33.973740 30410 leader_election.cc:288] T 00000000000000000000000000000000 P 865f1ba635c94f54925b7fd3919fbcd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51ce6a9e63ff44bf8e3947425a77cd7f (127.16.121.190:45555), f829a3a5cb534641a40e19edef77478d (127.16.121.189:36583)
I20260928 22:51:33.974238 30228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "865f1ba635c94f54925b7fd3919fbcd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51ce6a9e63ff44bf8e3947425a77cd7f" is_pre_election: true
I20260928 22:51:33.974429 30228 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 51ce6a9e63ff44bf8e3947425a77cd7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 865f1ba635c94f54925b7fd3919fbcd0 in term 0.
I20260928 22:51:33.974696 30353 leader_election.cc:302] T 00000000000000000000000000000000 P 865f1ba635c94f54925b7fd3919fbcd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51ce6a9e63ff44bf8e3947425a77cd7f, 865f1ba635c94f54925b7fd3919fbcd0; no voters:
I20260928 22:51:33.975234 30410 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 865f1ba635c94f54925b7fd3919fbcd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:51:33.975345 30410 raft_consensus.cc:482] T 00000000000000000000000000000000 P 865f1ba635c94f54925b7fd3919fbcd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:51:33.975375 30410 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 865f1ba635c94f54925b7fd3919fbcd0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:51:33.976788 30410 raft_consensus.cc:504] T 00000000000000000000000000000000 P 865f1ba635c94f54925b7fd3919fbcd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce6a9e63ff44bf8e3947425a77cd7f" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 45555 } } peers { permanent_uuid: "f829a3a5cb534641a40e19edef77478d" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 36583 } } peers { permanent_uuid: "865f1ba635c94f54925b7fd3919fbcd0" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 45567 } }
I20260928 22:51:33.976912 30410 leader_election.cc:288] T 00000000000000000000000000000000 P 865f1ba635c94f54925b7fd3919fbcd0 [CANDIDATE]: Term 1 election: Requested vote from peers 51ce6a9e63ff44bf8e3947425a77cd7f (127.16.121.190:45555), f829a3a5cb534641a40e19edef77478d (127.16.121.189:36583)
I20260928 22:51:33.977442 30228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "865f1ba635c94f54925b7fd3919fbcd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51ce6a9e63ff44bf8e3947425a77cd7f"
I20260928 22:51:33.977540 30228 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51ce6a9e63ff44bf8e3947425a77cd7f [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:51:33.978613 30228 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 51ce6a9e63ff44bf8e3947425a77cd7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 865f1ba635c94f54925b7fd3919fbcd0 in term 1.
I20260928 22:51:33.978830 30353 leader_election.cc:302] T 00000000000000000000000000000000 P 865f1ba635c94f54925b7fd3919fbcd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51ce6a9e63ff44bf8e3947425a77cd7f, 865f1ba635c94f54925b7fd3919fbcd0; no voters:
Sep 28 22:51:33 dist-test-slave-56vw krb5kdc[30173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635893, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.189@KRBTEST.COM for kudu/127.16.121.188@KRBTEST.COM
I20260928 22:51:33.980062 30410 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 865f1ba635c94f54925b7fd3919fbcd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:51:33.980228 30410 raft_consensus.cc:686] T 00000000000000000000000000000000 P 865f1ba635c94f54925b7fd3919fbcd0 [term 1 LEADER]: Becoming Leader. State: Replica: 865f1ba635c94f54925b7fd3919fbcd0, State: Running, Role: LEADER
I20260928 22:51:33.980327 30410 consensus_queue.cc:237] T 00000000000000000000000000000000 P 865f1ba635c94f54925b7fd3919fbcd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce6a9e63ff44bf8e3947425a77cd7f" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 45555 } } peers { permanent_uuid: "f829a3a5cb534641a40e19edef77478d" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 36583 } } peers { permanent_uuid: "865f1ba635c94f54925b7fd3919fbcd0" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 45567 } }
I20260928 22:51:33.980793 30410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 865f1ba635c94f54925b7fd3919fbcd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 865f1ba635c94f54925b7fd3919fbcd0. Latest consensus state: current_term: 1 leader_uuid: "865f1ba635c94f54925b7fd3919fbcd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce6a9e63ff44bf8e3947425a77cd7f" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 45555 } } peers { permanent_uuid: "f829a3a5cb534641a40e19edef77478d" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 36583 } } peers { permanent_uuid: "865f1ba635c94f54925b7fd3919fbcd0" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 45567 } } }
I20260928 22:51:33.980875 30410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 865f1ba635c94f54925b7fd3919fbcd0 [sys.catalog]: This master's current role is: LEADER
I20260928 22:51:33.981760 30447 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:51:33.982048 30447 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:51:33.982630 30331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:33.973376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:45567 (local address 127.0.0.1:37764)
0928 22:51:33.973478 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:33.973492 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 22:51:33.973541 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:51:33.974395 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:51:33.974396 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:51:33.974624 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:51:33.974743 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:33.974747 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:33.975851 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:33.975853 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:33.976687 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:33.976691 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:33.977234 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:33.977235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:33.977296 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:33.977831 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:51:33.977837 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:51:33.979521 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:51:33.981489 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:33.981490 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:33.981490 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:33.981706 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:33.981921 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:33.981922 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:33.981922 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:33.981981 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:33.982320 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:51:33.982321 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:51:33.982367 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 22:51:33.982430 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 22:51:33.982528 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260928 22:51:33.983146 30428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:33.973656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37764 (local address 127.16.121.188:45567)
0928 22:51:33.973862 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:33.973864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:33.973874 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:51:33.973901 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:33.973902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:33.973914 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:33.974021 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:51:33.974845 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:33.975739 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:33.976789 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:33.977111 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:33.977177 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:33.979610 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:51:33.979611 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:51:33.979612 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:51:33.979620 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:51:33.981396 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:33.981791 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:33.981792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:33.981793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:33.981848 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:33.982050 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:33.982051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:33.982051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:33.982230 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:51:33.982292 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:33.982957 (+ 665us) server_negotiation.cc:299] Negotiation successful
0928 22:51:33.983039 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182}
I20260928 22:51:33.984104 30228 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 51ce6a9e63ff44bf8e3947425a77cd7f [term 1 FOLLOWER]: Refusing update from remote peer 865f1ba635c94f54925b7fd3919fbcd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:51:33.984465 30445 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 865f1ba635c94f54925b7fd3919fbcd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51ce6a9e63ff44bf8e3947425a77cd7f" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 45555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:51:33.985153 30329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f829a3a5cb534641a40e19edef77478d: Bootstrap starting.
I20260928 22:51:33.985891 30329 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f829a3a5cb534641a40e19edef77478d: Neither blocks nor log segments found. Creating new log.
I20260928 22:51:33.985893 30432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51ce6a9e63ff44bf8e3947425a77cd7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 865f1ba635c94f54925b7fd3919fbcd0. Latest consensus state: current_term: 1 leader_uuid: "865f1ba635c94f54925b7fd3919fbcd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce6a9e63ff44bf8e3947425a77cd7f" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 45555 } } peers { permanent_uuid: "f829a3a5cb534641a40e19edef77478d" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 36583 } } peers { permanent_uuid: "865f1ba635c94f54925b7fd3919fbcd0" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 45567 } } }
I20260928 22:51:33.985970 30432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51ce6a9e63ff44bf8e3947425a77cd7f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:51:33.986228 30329 log.cc:824] T 00000000000000000000000000000000 P f829a3a5cb534641a40e19edef77478d: Log is configured to *not* fsync() on all Append() calls
I20260928 22:51:33.987416 30329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f829a3a5cb534641a40e19edef77478d: No bootstrap required, opened a new log
I20260928 22:51:33.988579 30445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 865f1ba635c94f54925b7fd3919fbcd0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "865f1ba635c94f54925b7fd3919fbcd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce6a9e63ff44bf8e3947425a77cd7f" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 45555 } } peers { permanent_uuid: "f829a3a5cb534641a40e19edef77478d" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 36583 } } peers { permanent_uuid: "865f1ba635c94f54925b7fd3919fbcd0" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 45567 } } }
I20260928 22:51:33.988673 30445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 865f1ba635c94f54925b7fd3919fbcd0 [sys.catalog]: This master's current role is: LEADER
I20260928 22:51:33.989094 30432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51ce6a9e63ff44bf8e3947425a77cd7f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "865f1ba635c94f54925b7fd3919fbcd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce6a9e63ff44bf8e3947425a77cd7f" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 45555 } } peers { permanent_uuid: "f829a3a5cb534641a40e19edef77478d" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 36583 } } peers { permanent_uuid: "865f1ba635c94f54925b7fd3919fbcd0" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 45567 } } }
I20260928 22:51:33.989169 30432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51ce6a9e63ff44bf8e3947425a77cd7f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:51:33.989413 30448 mvcc.cc:205] Tried to move back new op lower bound from 7334444621755162624 to 7334444621744189440. Current Snapshot: MvccSnapshot[applied={T|T < 7334444621755162624}]
I20260928 22:51:33.989669 30329 raft_consensus.cc:348] T 00000000000000000000000000000000 P f829a3a5cb534641a40e19edef77478d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce6a9e63ff44bf8e3947425a77cd7f" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 45555 } } peers { permanent_uuid: "f829a3a5cb534641a40e19edef77478d" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 36583 } } peers { permanent_uuid: "865f1ba635c94f54925b7fd3919fbcd0" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 45567 } }
I20260928 22:51:33.989800 30329 raft_consensus.cc:374] T 00000000000000000000000000000000 P f829a3a5cb534641a40e19edef77478d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:51:33.989836 30329 raft_consensus.cc:729] T 00000000000000000000000000000000 P f829a3a5cb534641a40e19edef77478d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f829a3a5cb534641a40e19edef77478d, State: Initialized, Role: FOLLOWER
I20260928 22:51:33.989955 30329 consensus_queue.cc:260] T 00000000000000000000000000000000 P f829a3a5cb534641a40e19edef77478d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce6a9e63ff44bf8e3947425a77cd7f" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 45555 } } peers { permanent_uuid: "f829a3a5cb534641a40e19edef77478d" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 36583 } } peers { permanent_uuid: "865f1ba635c94f54925b7fd3919fbcd0" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 45567 } }
I20260928 22:51:33.990147 30449 mvcc.cc:205] Tried to move back new op lower bound from 7334444621755162624 to 7334444621744189440. Current Snapshot: MvccSnapshot[applied={T|T < 7334444621755162624}]
I20260928 22:51:33.990274 30329 sys_catalog.cc:564] T 00000000000000000000000000000000 P f829a3a5cb534641a40e19edef77478d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:51:33.990361 30302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "865f1ba635c94f54925b7fd3919fbcd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f829a3a5cb534641a40e19edef77478d"
I20260928 22:51:33.990411 30303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "865f1ba635c94f54925b7fd3919fbcd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f829a3a5cb534641a40e19edef77478d" is_pre_election: true
I20260928 22:51:33.990499 30302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f829a3a5cb534641a40e19edef77478d [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:51:33.991525 30447 catalog_manager.cc:1347] Generated new cluster ID: 951b65efb66041f08ab9a10af2e74590
I20260928 22:51:33.991581 30447 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:51:33.991678 30302 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f829a3a5cb534641a40e19edef77478d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 865f1ba635c94f54925b7fd3919fbcd0 in term 1.
I20260928 22:51:33.991801 30454 sys_catalog.cc:455] T 00000000000000000000000000000000 P f829a3a5cb534641a40e19edef77478d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce6a9e63ff44bf8e3947425a77cd7f" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 45555 } } peers { permanent_uuid: "f829a3a5cb534641a40e19edef77478d" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 36583 } } peers { permanent_uuid: "865f1ba635c94f54925b7fd3919fbcd0" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 45567 } } }
I20260928 22:51:33.991858 30301 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f829a3a5cb534641a40e19edef77478d [term 1 FOLLOWER]: Refusing update from remote peer 865f1ba635c94f54925b7fd3919fbcd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:51:33.991886 30454 sys_catalog.cc:458] T 00000000000000000000000000000000 P f829a3a5cb534641a40e19edef77478d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:51:33.992372 30410 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 865f1ba635c94f54925b7fd3919fbcd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f829a3a5cb534641a40e19edef77478d" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 36583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:51:33.993978 30454 sys_catalog.cc:455] T 00000000000000000000000000000000 P f829a3a5cb534641a40e19edef77478d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 865f1ba635c94f54925b7fd3919fbcd0. Latest consensus state: current_term: 1 leader_uuid: "865f1ba635c94f54925b7fd3919fbcd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce6a9e63ff44bf8e3947425a77cd7f" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 45555 } } peers { permanent_uuid: "f829a3a5cb534641a40e19edef77478d" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 36583 } } peers { permanent_uuid: "865f1ba635c94f54925b7fd3919fbcd0" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 45567 } } }
I20260928 22:51:33.994051 30454 sys_catalog.cc:458] T 00000000000000000000000000000000 P f829a3a5cb534641a40e19edef77478d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:51:33.994079 30454 sys_catalog.cc:455] T 00000000000000000000000000000000 P f829a3a5cb534641a40e19edef77478d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "865f1ba635c94f54925b7fd3919fbcd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce6a9e63ff44bf8e3947425a77cd7f" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 45555 } } peers { permanent_uuid: "f829a3a5cb534641a40e19edef77478d" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 36583 } } peers { permanent_uuid: "865f1ba635c94f54925b7fd3919fbcd0" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 45567 } } }
I20260928 22:51:33.994109 30454 sys_catalog.cc:458] T 00000000000000000000000000000000 P f829a3a5cb534641a40e19edef77478d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:51:33.995944 30466 mvcc.cc:205] Tried to move back new op lower bound from 7334444621755162624 to 7334444621744189440. Current Snapshot: MvccSnapshot[applied={T|T < 7334444621755162624}]
I20260928 22:51:33.999679 30465 catalog_manager.cc:1259] Loaded cluster ID: 951b65efb66041f08ab9a10af2e74590
I20260928 22:51:33.999763 30465 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f829a3a5cb534641a40e19edef77478d: loading cluster ID for follower catalog manager: success
W20260928 22:51:33.999876 30465 catalog_manager.cc:1576] T 00000000000000000000000000000000 P f829a3a5cb534641a40e19edef77478d: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260928 22:51:33.999949 30465 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:51:34.019387 30410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 865f1ba635c94f54925b7fd3919fbcd0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "865f1ba635c94f54925b7fd3919fbcd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce6a9e63ff44bf8e3947425a77cd7f" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 45555 } } peers { permanent_uuid: "f829a3a5cb534641a40e19edef77478d" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 36583 } } peers { permanent_uuid: "865f1ba635c94f54925b7fd3919fbcd0" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 45567 } } }
I20260928 22:51:34.019538 30410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 865f1ba635c94f54925b7fd3919fbcd0 [sys.catalog]: This master's current role is: LEADER
I20260928 22:51:34.021093 30447 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:51:34.021895 30447 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:51:34.031271 30447 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 865f1ba635c94f54925b7fd3919fbcd0: Generated new TSK 0
I20260928 22:51:34.031561 30447 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:51:34.946907 30443 catalog_manager.cc:1259] Loaded cluster ID: 951b65efb66041f08ab9a10af2e74590
I20260928 22:51:34.946986 30443 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 51ce6a9e63ff44bf8e3947425a77cd7f: loading cluster ID for follower catalog manager: success
I20260928 22:51:34.947852 30443 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 51ce6a9e63ff44bf8e3947425a77cd7f: acquiring CA information for follower catalog manager: success
I20260928 22:51:34.948118 30443 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 51ce6a9e63ff44bf8e3947425a77cd7f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 22:51:35.001139 30465 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f829a3a5cb534641a40e19edef77478d: acquiring CA information for follower catalog manager: success
I20260928 22:51:35.001552 30465 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f829a3a5cb534641a40e19edef77478d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 22:51:35.087419 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 30186
I20260928 22:51:35.093550 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 30261
I20260928 22:51:35.099102 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 30336
2026-09-28T22:51:35Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2813 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635733329745-16870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:51:35 dist-test-slave-56vw krb5kdc[30478](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:51:35 dist-test-slave-56vw krb5kdc[30478](info): set up 2 sockets
Sep 28 22:51:35 dist-test-slave-56vw krb5kdc[30478](info): commencing operation
krb5kdc: starting...
W20260928 22:51:36.151307 16870 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Sep 28 22:51:36 dist-test-slave-56vw krb5kdc[30478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:51:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:51:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.121.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
I20260928 22:51:36.278980 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:45707
--webserver_interface=127.16.121.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:42665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:45707
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:51:36.361789 30491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:51:36.361997 30491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:51:36.362021 30491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:51:36.363178 30491 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:51:36.363215 30491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:51:36.363229 30491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:51:36.363242 30491 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:51:36.363255 30491 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:51:36.364817 30491 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:42665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:45707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:45707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.121.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:51:36.365016 30491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:51:36.365195 30491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:51:36.370225 30499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:51:36.370232 30496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:51:36.370443 30491 server_base.cc:1034] running on GCE node
W20260928 22:51:36.370232 30497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:51:36.370770 30491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:51:36.371160 30491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:51:36.372346 30491 hybrid_clock.cc:648] HybridClock initialized: now 1790635896372323 us; error 40 us; skew 500 ppm
Sep 28 22:51:36 dist-test-slave-56vw krb5kdc[30478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635896, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:51:36.375289 30491 init.cc:377] Logged in from keytab as kudu/127.16.121.190@KRBTEST.COM (short username kudu)
I20260928 22:51:36.375818 30491 webserver.cc:460] Webserver started at http://127.16.121.190:40167/ using document root <none> and password file <none>
I20260928 22:51:36.376083 30491 fs_manager.cc:359] Metadata directory not provided
I20260928 22:51:36.376129 30491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:51:36.376282 30491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:51:36.377403 30491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "7854e0b23a784dd78aa4e8553a356cbd"
format_stamp: "Formatted at 2026-09-28 22:51:36 on dist-test-slave-56vw"
server_key: "33c751306963eb74d7a5209ce57ce6c0"
server_key_iv: "0ea5007dc961bac0291d417efa909d79"
server_key_version: "encryptionkey@0"
I20260928 22:51:36.377779 30491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "7854e0b23a784dd78aa4e8553a356cbd"
format_stamp: "Formatted at 2026-09-28 22:51:36 on dist-test-slave-56vw"
server_key: "33c751306963eb74d7a5209ce57ce6c0"
server_key_iv: "0ea5007dc961bac0291d417efa909d79"
server_key_version: "encryptionkey@0"
I20260928 22:51:36.379249 30491 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:51:36.380113 30506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:36.380298 30491 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:51:36.380405 30491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "7854e0b23a784dd78aa4e8553a356cbd"
format_stamp: "Formatted at 2026-09-28 22:51:36 on dist-test-slave-56vw"
server_key: "33c751306963eb74d7a5209ce57ce6c0"
server_key_iv: "0ea5007dc961bac0291d417efa909d79"
server_key_version: "encryptionkey@0"
I20260928 22:51:36.380494 30491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:51:36.397151 30491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:51:36.399544 30491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:51:36.399757 30491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:51:36.403980 30491 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.190:45707
I20260928 22:51:36.404014 30558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.190:45707 every 8 connection(s)
I20260928 22:51:36.404397 30491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:51:36.405443 30559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:51:36.408004 30559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7854e0b23a784dd78aa4e8553a356cbd: Bootstrap starting.
I20260928 22:51:36.408627 30559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7854e0b23a784dd78aa4e8553a356cbd: Neither blocks nor log segments found. Creating new log.
I20260928 22:51:36.408939 30559 log.cc:824] T 00000000000000000000000000000000 P 7854e0b23a784dd78aa4e8553a356cbd: Log is configured to *not* fsync() on all Append() calls
I20260928 22:51:36.409699 30559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7854e0b23a784dd78aa4e8553a356cbd: No bootstrap required, opened a new log
I20260928 22:51:36.411492 30559 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7854e0b23a784dd78aa4e8553a356cbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7854e0b23a784dd78aa4e8553a356cbd" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 45707 } }
I20260928 22:51:36.411636 30559 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7854e0b23a784dd78aa4e8553a356cbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:51:36.411659 30559 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7854e0b23a784dd78aa4e8553a356cbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7854e0b23a784dd78aa4e8553a356cbd, State: Initialized, Role: FOLLOWER
I20260928 22:51:36.411832 30559 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7854e0b23a784dd78aa4e8553a356cbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7854e0b23a784dd78aa4e8553a356cbd" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 45707 } }
I20260928 22:51:36.411899 30559 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7854e0b23a784dd78aa4e8553a356cbd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:51:36.411921 30559 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7854e0b23a784dd78aa4e8553a356cbd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:51:36.411970 30559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7854e0b23a784dd78aa4e8553a356cbd [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:51:36.412532 30559 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7854e0b23a784dd78aa4e8553a356cbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7854e0b23a784dd78aa4e8553a356cbd" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 45707 } }
I20260928 22:51:36.412616 30559 leader_election.cc:302] T 00000000000000000000000000000000 P 7854e0b23a784dd78aa4e8553a356cbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7854e0b23a784dd78aa4e8553a356cbd; no voters:
I20260928 22:51:36.412803 30559 leader_election.cc:288] T 00000000000000000000000000000000 P 7854e0b23a784dd78aa4e8553a356cbd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:51:36.412937 30562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7854e0b23a784dd78aa4e8553a356cbd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:51:36.413113 30559 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7854e0b23a784dd78aa4e8553a356cbd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:51:36.413159 30562 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7854e0b23a784dd78aa4e8553a356cbd [term 1 LEADER]: Becoming Leader. State: Replica: 7854e0b23a784dd78aa4e8553a356cbd, State: Running, Role: LEADER
I20260928 22:51:36.413348 30562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7854e0b23a784dd78aa4e8553a356cbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7854e0b23a784dd78aa4e8553a356cbd" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 45707 } }
I20260928 22:51:36.413949 30564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7854e0b23a784dd78aa4e8553a356cbd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7854e0b23a784dd78aa4e8553a356cbd. Latest consensus state: current_term: 1 leader_uuid: "7854e0b23a784dd78aa4e8553a356cbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7854e0b23a784dd78aa4e8553a356cbd" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 45707 } } }
I20260928 22:51:36.414417 30564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7854e0b23a784dd78aa4e8553a356cbd [sys.catalog]: This master's current role is: LEADER
I20260928 22:51:36.414490 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 30491
I20260928 22:51:36.414316 30563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7854e0b23a784dd78aa4e8553a356cbd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7854e0b23a784dd78aa4e8553a356cbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7854e0b23a784dd78aa4e8553a356cbd" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 45707 } } }
I20260928 22:51:36.414594 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:51:36.414788 16870 external_mini_cluster.cc:1428] Setting key 19ed7b1a4349c15efd8f0ab6cf56ccea
I20260928 22:51:36.414691 30563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7854e0b23a784dd78aa4e8553a356cbd [sys.catalog]: This master's current role is: LEADER
W20260928 22:51:36.415292 30578 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7854e0b23a784dd78aa4e8553a356cbd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:51:36.415349 30578 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:51:36.415395 30576 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:51:36.415792 30576 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:51:36.419524 30576 catalog_manager.cc:1347] Generated new cluster ID: 4900d103be06494b8b33503e4e4d47a6
I20260928 22:51:36.419592 30576 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 22:51:36 dist-test-slave-56vw krb5kdc[30478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:51:36.430245 30576 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:51:36.430301 30581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:36.415503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53794 (local address 127.16.121.190:45707)
0928 22:51:36.415711 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:36.415722 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:36.416611 (+ 889us) server_negotiation.cc:407] Connection header received
0928 22:51:36.416745 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:36.416754 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:36.416829 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:36.416991 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:51:36.417735 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:36.418800 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:36.423530 (+ 4730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:36.424051 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:36.424189 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:36.426633 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:51:36.426643 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:51:36.426643 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:51:36.426670 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:51:36.428522 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:36.428931 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:36.428938 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:36.428939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:36.429038 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:36.429268 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:36.429269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:36.429269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:36.429531 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:51:36.429697 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:36.429814 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 22:51:36.429967 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":77,"threads_started":1}
I20260928 22:51:36.430943 30576 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:51:36.431206 16870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:51:36.439021 30576 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7854e0b23a784dd78aa4e8553a356cbd: Generated new TSK 0
I20260928 22:51:36.439294 30576 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:51:36.485384 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 30491
2026-09-28T22:51:36Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1384 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:51:36 dist-test-slave-56vw krb5kdc[30586](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:51:36 dist-test-slave-56vw krb5kdc[30586](info): set up 2 sockets
Sep 28 22:51:36 dist-test-slave-56vw krb5kdc[30586](info): commencing operation
krb5kdc: starting...
W20260928 22:51:37.555303 16870 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Sep 28 22:51:37 dist-test-slave-56vw krb5kdc[30586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:51:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:51:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.121.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
I20260928 22:51:37.682834 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:37803
--webserver_interface=127.16.121.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:36021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:37803
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:51:37.763916 30599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:51:37.764263 30599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:51:37.764317 30599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:51:37.766530 30599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:51:37.766598 30599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:51:37.766625 30599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:51:37.766660 30599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:51:37.766707 30599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:51:37.769222 30599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:36021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:37803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:37803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.121.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:51:37.769506 30599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:51:37.769801 30599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:51:37.776333 30607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:51:37.776423 30605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:51:37.776325 30604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:51:37.776566 30599 server_base.cc:1034] running on GCE node
I20260928 22:51:37.776793 30599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:51:37.777117 30599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:51:37.778278 30599 hybrid_clock.cc:648] HybridClock initialized: now 1790635897778249 us; error 40 us; skew 500 ppm
Sep 28 22:51:37 dist-test-slave-56vw krb5kdc[30586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635897, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:51:37.781001 30599 init.cc:377] Logged in from keytab as kudu/127.16.121.190@KRBTEST.COM (short username kudu)
I20260928 22:51:37.781522 30599 webserver.cc:460] Webserver started at http://127.16.121.190:39081/ using document root <none> and password file <none>
I20260928 22:51:37.781756 30599 fs_manager.cc:359] Metadata directory not provided
I20260928 22:51:37.781810 30599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:51:37.781945 30599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:51:37.783123 30599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "2fd24b851da0411193bead83d7411447"
format_stamp: "Formatted at 2026-09-28 22:51:37 on dist-test-slave-56vw"
server_key: "93823709de34f9f37ec9d781eb76d7b7"
server_key_iv: "a281b7f64ec9c6a39a0ca05d6f29e18b"
server_key_version: "encryptionkey@0"
I20260928 22:51:37.783495 30599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "2fd24b851da0411193bead83d7411447"
format_stamp: "Formatted at 2026-09-28 22:51:37 on dist-test-slave-56vw"
server_key: "93823709de34f9f37ec9d781eb76d7b7"
server_key_iv: "a281b7f64ec9c6a39a0ca05d6f29e18b"
server_key_version: "encryptionkey@0"
I20260928 22:51:37.784893 30599 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:51:37.785647 30614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:37.785866 30599 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:51:37.785972 30599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "2fd24b851da0411193bead83d7411447"
format_stamp: "Formatted at 2026-09-28 22:51:37 on dist-test-slave-56vw"
server_key: "93823709de34f9f37ec9d781eb76d7b7"
server_key_iv: "a281b7f64ec9c6a39a0ca05d6f29e18b"
server_key_version: "encryptionkey@0"
I20260928 22:51:37.786055 30599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:51:37.801975 30599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:51:37.804370 30599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:51:37.804554 30599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:51:37.808840 30599 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.190:37803
I20260928 22:51:37.808882 30666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.190:37803 every 8 connection(s)
I20260928 22:51:37.809238 30599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:51:37.810225 30667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:51:37.812633 30667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447: Bootstrap starting.
I20260928 22:51:37.813228 30667 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447: Neither blocks nor log segments found. Creating new log.
I20260928 22:51:37.813516 30667 log.cc:824] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447: Log is configured to *not* fsync() on all Append() calls
I20260928 22:51:37.814352 30667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447: No bootstrap required, opened a new log
I20260928 22:51:37.816097 30667 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fd24b851da0411193bead83d7411447" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37803 } }
I20260928 22:51:37.816243 30667 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:51:37.816277 30667 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fd24b851da0411193bead83d7411447, State: Initialized, Role: FOLLOWER
I20260928 22:51:37.816435 30667 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fd24b851da0411193bead83d7411447" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37803 } }
I20260928 22:51:37.816500 30667 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:51:37.816541 30667 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:51:37.816593 30667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:51:37.817430 30667 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fd24b851da0411193bead83d7411447" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37803 } }
I20260928 22:51:37.817523 30667 leader_election.cc:302] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fd24b851da0411193bead83d7411447; no voters:
I20260928 22:51:37.817691 30667 leader_election.cc:288] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:51:37.817904 30670 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:51:37.817955 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 30599
I20260928 22:51:37.817975 30667 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:51:37.818074 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:51:37.818136 30670 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [term 1 LEADER]: Becoming Leader. State: Replica: 2fd24b851da0411193bead83d7411447, State: Running, Role: LEADER
I20260928 22:51:37.818252 30670 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fd24b851da0411193bead83d7411447" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37803 } }
I20260928 22:51:37.818382 16870 external_mini_cluster.cc:1428] Setting key b9a81d23f41ed3d954e3fdabc15cfd9d
I20260928 22:51:37.819108 30672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fd24b851da0411193bead83d7411447. Latest consensus state: current_term: 1 leader_uuid: "2fd24b851da0411193bead83d7411447" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fd24b851da0411193bead83d7411447" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37803 } } }
I20260928 22:51:37.819227 30672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [sys.catalog]: This master's current role is: LEADER
I20260928 22:51:37.819664 30671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2fd24b851da0411193bead83d7411447" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fd24b851da0411193bead83d7411447" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37803 } } }
I20260928 22:51:37.819746 30671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [sys.catalog]: This master's current role is: LEADER
W20260928 22:51:37.819939 30688 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:51:37.819988 30688 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:51:37.820077 30687 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:51:37.820303 30687 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:51:37.821867 30687 catalog_manager.cc:1347] Generated new cluster ID: a02d1fff1901438a888b8cd9d3c64119
I20260928 22:51:37.821923 30687 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 22:51:37 dist-test-slave-56vw krb5kdc[30586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:51:37.830477 30680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:37.818883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35662 (local address 127.16.121.190:37803)
0928 22:51:37.819090 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:37.819101 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:37.819416 (+ 315us) server_negotiation.cc:407] Connection header received
0928 22:51:37.819634 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:37.819643 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:37.819706 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:37.819838 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:51:37.820771 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:37.822127 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:37.823197 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:37.823840 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:37.823954 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:37.826798 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:51:37.826802 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:51:37.826803 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:51:37.826822 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:51:37.828781 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:37.829200 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:37.829203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:37.829203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:37.829323 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:37.829537 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:37.829538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:37.829538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:37.829848 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:51:37.829994 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:37.830128 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 22:51:37.830261 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":79,"threads_started":1}
I20260928 22:51:37.838555 30687 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:51:37.839334 30687 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:51:37.853622 30687 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447: Generated new TSK 0
I20260928 22:51:37.853878 30687 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.121.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab.
Entry for principal kudu/127.16.121.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab.
I20260928 22:51:37.890748 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.129:0
--local_ip_for_outbound_sockets=127.16.121.129
--webserver_interface=127.16.121.129
--webserver_port=0
--tserver_master_addrs=127.16.121.190:37803
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab
--principal=kudu/127.16.121.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:36021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:51:37.982223 30693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:51:37.982489 30693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:51:37.982537 30693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:51:37.983978 30693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:51:37.984051 30693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:51:37.984112 30693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.129
I20260928 22:51:37.985777 30693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:36021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab
--principal=kudu/127.16.121.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.121.129
--webserver_port=0
--tserver_master_addrs=127.16.121.190:37803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.129
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:51:37.986080 30693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:51:37.986342 30693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:51:37.992333 30698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:51:37.992412 30693 server_base.cc:1034] running on GCE node
W20260928 22:51:37.992403 30701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:51:37.992337 30699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:51:37.992744 30693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:51:37.992952 30693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:51:37.994099 30693 hybrid_clock.cc:648] HybridClock initialized: now 1790635897994081 us; error 43 us; skew 500 ppm
Sep 28 22:51:37 dist-test-slave-56vw krb5kdc[30586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635897, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:51:37.996881 30693 init.cc:377] Logged in from keytab as kudu/127.16.121.129@KRBTEST.COM (short username kudu)
I20260928 22:51:37.997349 30693 webserver.cc:460] Webserver started at http://127.16.121.129:46821/ using document root <none> and password file <none>
I20260928 22:51:37.997604 30693 fs_manager.cc:359] Metadata directory not provided
I20260928 22:51:37.997676 30693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:51:37.997826 30693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:51:37.998883 30693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-0/data/instance:
uuid: "86d13097cb5348d2a58f4c202689f82e"
format_stamp: "Formatted at 2026-09-28 22:51:37 on dist-test-slave-56vw"
server_key: "d1ab9569fc2b4e447e0579f94ac0eb03"
server_key_iv: "06a6de36040dba02659a2d0aa34e60a8"
server_key_version: "encryptionkey@0"
I20260928 22:51:37.999274 30693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance:
uuid: "86d13097cb5348d2a58f4c202689f82e"
format_stamp: "Formatted at 2026-09-28 22:51:37 on dist-test-slave-56vw"
server_key: "d1ab9569fc2b4e447e0579f94ac0eb03"
server_key_iv: "06a6de36040dba02659a2d0aa34e60a8"
server_key_version: "encryptionkey@0"
I20260928 22:51:38.000622 30693 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:51:38.001386 30708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:38.001554 30693 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:51:38.001657 30693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-0/wal
uuid: "86d13097cb5348d2a58f4c202689f82e"
format_stamp: "Formatted at 2026-09-28 22:51:37 on dist-test-slave-56vw"
server_key: "d1ab9569fc2b4e447e0579f94ac0eb03"
server_key_iv: "06a6de36040dba02659a2d0aa34e60a8"
server_key_version: "encryptionkey@0"
I20260928 22:51:38.001734 30693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:51:38.010968 30693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:51:38.012977 30693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:51:38.013152 30693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:51:38.013393 30693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:51:38.013720 30693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:51:38.013777 30693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:38.013811 30693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:51:38.013839 30693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:38.020541 30693 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.129:36367
I20260928 22:51:38.020586 30821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.129:36367 every 8 connection(s)
I20260928 22:51:38.020968 30693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
I20260928 22:51:38.026335 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 30693
I20260928 22:51:38.026466 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance
I20260928 22:51:38.026716 16870 external_mini_cluster.cc:1428] Setting key fb81bf43d601646e542f53d360eac129
Sep 28 22:51:38 dist-test-slave-56vw krb5kdc[30586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635897, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:51:38.032330 30680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:38.021825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:53335 (local address 127.16.121.190:37803)
0928 22:51:38.021900 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:38.021921 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:38.022141 (+ 220us) server_negotiation.cc:407] Connection header received
0928 22:51:38.022870 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:38.022872 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:38.022892 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:38.022971 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:51:38.024066 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:38.024595 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:38.025693 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:38.025914 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:38.025996 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:38.028831 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:51:38.028833 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:51:38.028833 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:51:38.028841 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:51:38.030512 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:38.031107 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:38.031109 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:38.031109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:38.031176 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:38.031484 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:38.031485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:38.031485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:38.031621 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:51:38.031710 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:38.032106 (+ 396us) server_negotiation.cc:299] Negotiation successful
0928 22:51:38.032227 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 22:51:38.032640 30824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:38.021795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.129:53335)
0928 22:51:38.022000 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:38.022041 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 22:51:38.022659 (+ 618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:51:38.023090 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:51:38.023094 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:51:38.023714 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:51:38.023944 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:38.023949 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:38.024706 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:38.024707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:38.025583 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:38.025585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:38.026019 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:38.026020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:38.026211 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:38.026996 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:51:38.027005 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:51:38.028716 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:51:38.030658 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:38.030666 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:38.030669 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:38.030956 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:38.031274 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:38.031275 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:38.031275 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:38.031406 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:38.031793 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:51:38.031801 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:51:38.031945 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 22:51:38.032071 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 22:51:38.032261 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":93,"threads_started":1}
I20260928 22:51:38.033283 30822 heartbeater.cc:347] Connected to a master server at 127.16.121.190:37803
I20260928 22:51:38.033414 30822 heartbeater.cc:464] Registering TS with master...
I20260928 22:51:38.033622 30822 heartbeater.cc:511] Master 127.16.121.190:37803 requested a full tablet report, sending...
I20260928 22:51:38.034296 30631 ts_manager.cc:194] Registered new tserver with Master: 86d13097cb5348d2a58f4c202689f82e (127.16.121.129:36367)
WARNING: no policy specified for kudu/127.16.121.130@KRBTEST.COM; defaulting to no policy
I20260928 22:51:38.035126 30631 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.129@KRBTEST.COM'} at 127.16.121.129:53335
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab.
Entry for principal kudu/127.16.121.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab.
I20260928 22:51:38.054795 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.130:0
--local_ip_for_outbound_sockets=127.16.121.130
--webserver_interface=127.16.121.130
--webserver_port=0
--tserver_master_addrs=127.16.121.190:37803
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab
--principal=kudu/127.16.121.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:36021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:51:38.138271 30827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:51:38.138499 30827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:51:38.138523 30827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:51:38.140266 30827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:51:38.140321 30827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:51:38.140348 30827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.130
I20260928 22:51:38.141841 30827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:36021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab
--principal=kudu/127.16.121.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.121.130
--webserver_port=0
--tserver_master_addrs=127.16.121.190:37803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.130
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:51:38.142102 30827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:51:38.142343 30827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:51:38.147782 30832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:51:38.147760 30835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:51:38.147765 30827 server_base.cc:1034] running on GCE node
W20260928 22:51:38.147751 30833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:51:38.148288 30827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:51:38.148586 30827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:51:38.149871 30827 hybrid_clock.cc:648] HybridClock initialized: now 1790635898149855 us; error 25 us; skew 500 ppm
Sep 28 22:51:38 dist-test-slave-56vw krb5kdc[30586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635898, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:51:38.152673 30827 init.cc:377] Logged in from keytab as kudu/127.16.121.130@KRBTEST.COM (short username kudu)
I20260928 22:51:38.153188 30827 webserver.cc:460] Webserver started at http://127.16.121.130:42105/ using document root <none> and password file <none>
I20260928 22:51:38.153486 30827 fs_manager.cc:359] Metadata directory not provided
I20260928 22:51:38.153563 30827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:51:38.153726 30827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:51:38.155104 30827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-1/data/instance:
uuid: "90486aa702d140f19e71be00c5088990"
format_stamp: "Formatted at 2026-09-28 22:51:38 on dist-test-slave-56vw"
server_key: "c5b4f7ce9c47173fe00ad0a32ca73011"
server_key_iv: "627c6ae39d4c7a429c96adddce7d38cd"
server_key_version: "encryptionkey@0"
I20260928 22:51:38.155520 30827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance:
uuid: "90486aa702d140f19e71be00c5088990"
format_stamp: "Formatted at 2026-09-28 22:51:38 on dist-test-slave-56vw"
server_key: "c5b4f7ce9c47173fe00ad0a32ca73011"
server_key_iv: "627c6ae39d4c7a429c96adddce7d38cd"
server_key_version: "encryptionkey@0"
I20260928 22:51:38.157009 30827 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:51:38.157908 30842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:38.158120 30827 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:51:38.158234 30827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-1/wal
uuid: "90486aa702d140f19e71be00c5088990"
format_stamp: "Formatted at 2026-09-28 22:51:38 on dist-test-slave-56vw"
server_key: "c5b4f7ce9c47173fe00ad0a32ca73011"
server_key_iv: "627c6ae39d4c7a429c96adddce7d38cd"
server_key_version: "encryptionkey@0"
I20260928 22:51:38.158326 30827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:51:38.194352 30827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:51:38.197090 30827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:51:38.197321 30827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:51:38.197609 30827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:51:38.198001 30827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:51:38.198063 30827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:38.198102 30827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:51:38.198135 30827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:38.204742 30827 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.130:41643
I20260928 22:51:38.204785 30955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.130:41643 every 8 connection(s)
I20260928 22:51:38.205188 30827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
I20260928 22:51:38.210431 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 30827
I20260928 22:51:38.210526 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance
I20260928 22:51:38.210731 16870 external_mini_cluster.cc:1428] Setting key ef9edde4b66d3d15ca20fa89068d1a3b
Sep 28 22:51:38 dist-test-slave-56vw krb5kdc[30586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635898, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.130@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:51:38.215956 30680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:38.206120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:36367 (local address 127.16.121.190:37803)
0928 22:51:38.206200 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:38.206202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:38.206424 (+ 222us) server_negotiation.cc:407] Connection header received
0928 22:51:38.207083 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:38.207085 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:38.207102 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:38.207180 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:51:38.208264 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:38.208837 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:38.209968 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:38.210235 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:38.210317 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:38.212966 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:51:38.212968 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:51:38.212968 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:51:38.212982 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:51:38.214476 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:38.214925 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:38.214927 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:38.214927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:38.214987 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:38.215265 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:38.215266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:38.215266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:38.215392 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:51:38.215469 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:38.215759 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 22:51:38.215857 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 22:51:38.216183 30958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:38.206103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.130:36367)
0928 22:51:38.206285 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:38.206327 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 22:51:38.206915 (+ 588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:51:38.207295 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:51:38.207300 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:51:38.207920 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:51:38.208128 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:38.208133 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:38.208955 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:38.208956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:38.209864 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:38.209866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:38.210364 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:38.210365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:38.210522 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:38.211243 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:51:38.211251 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:51:38.212775 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:51:38.214573 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:38.214580 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:38.214583 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:38.214830 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:38.215080 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:38.215080 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:38.215081 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:38.215195 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:38.215460 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:51:38.215467 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:51:38.215628 (+ 161us) client_negotiation.cc:769] Sending connection context
0928 22:51:38.215740 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 22:51:38.215925 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":70,"threads_started":1}
I20260928 22:51:38.216737 30956 heartbeater.cc:347] Connected to a master server at 127.16.121.190:37803
I20260928 22:51:38.216835 30956 heartbeater.cc:464] Registering TS with master...
I20260928 22:51:38.217038 30956 heartbeater.cc:511] Master 127.16.121.190:37803 requested a full tablet report, sending...
I20260928 22:51:38.217471 30631 ts_manager.cc:194] Registered new tserver with Master: 90486aa702d140f19e71be00c5088990 (127.16.121.130:41643)
I20260928 22:51:38.218025 30631 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.130@KRBTEST.COM'} at 127.16.121.130:36367
WARNING: no policy specified for kudu/127.16.121.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab.
Entry for principal kudu/127.16.121.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab.
I20260928 22:51:38.239475 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.131:0
--local_ip_for_outbound_sockets=127.16.121.131
--webserver_interface=127.16.121.131
--webserver_port=0
--tserver_master_addrs=127.16.121.190:37803
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab
--principal=kudu/127.16.121.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:36021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:51:38.323658 30961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:51:38.323879 30961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:51:38.323904 30961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:51:38.325102 30961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:51:38.325146 30961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:51:38.325174 30961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.131
I20260928 22:51:38.326606 30961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:36021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab
--principal=kudu/127.16.121.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.121.131
--webserver_port=0
--tserver_master_addrs=127.16.121.190:37803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.131
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:51:38.326833 30961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:51:38.327025 30961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:51:38.332192 30966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:51:38.332188 30969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:51:38.332198 30967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:51:38.332450 30961 server_base.cc:1034] running on GCE node
I20260928 22:51:38.332650 30961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:51:38.332891 30961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:51:38.334060 30961 hybrid_clock.cc:648] HybridClock initialized: now 1790635898334048 us; error 30 us; skew 500 ppm
Sep 28 22:51:38 dist-test-slave-56vw krb5kdc[30586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635898, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:51:38.336576 30961 init.cc:377] Logged in from keytab as kudu/127.16.121.131@KRBTEST.COM (short username kudu)
I20260928 22:51:38.337039 30961 webserver.cc:460] Webserver started at http://127.16.121.131:45171/ using document root <none> and password file <none>
I20260928 22:51:38.337282 30961 fs_manager.cc:359] Metadata directory not provided
I20260928 22:51:38.337353 30961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:51:38.337500 30961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:51:38.338559 30961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-2/data/instance:
uuid: "87072ae366cd40b48beee55f28afb7b9"
format_stamp: "Formatted at 2026-09-28 22:51:38 on dist-test-slave-56vw"
server_key: "731b56b30449e31d337fb18fa7a0f486"
server_key_iv: "840d244dc374d7f4e039b7ed8a04cc69"
server_key_version: "encryptionkey@0"
I20260928 22:51:38.338927 30961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance:
uuid: "87072ae366cd40b48beee55f28afb7b9"
format_stamp: "Formatted at 2026-09-28 22:51:38 on dist-test-slave-56vw"
server_key: "731b56b30449e31d337fb18fa7a0f486"
server_key_iv: "840d244dc374d7f4e039b7ed8a04cc69"
server_key_version: "encryptionkey@0"
I20260928 22:51:38.340284 30961 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:51:38.340993 30976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:38.341176 30961 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:51:38.341271 30961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-2/wal
uuid: "87072ae366cd40b48beee55f28afb7b9"
format_stamp: "Formatted at 2026-09-28 22:51:38 on dist-test-slave-56vw"
server_key: "731b56b30449e31d337fb18fa7a0f486"
server_key_iv: "840d244dc374d7f4e039b7ed8a04cc69"
server_key_version: "encryptionkey@0"
I20260928 22:51:38.341355 30961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:51:38.358618 30961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:51:38.360628 30961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:51:38.360792 30961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:51:38.361033 30961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:51:38.361368 30961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:51:38.361423 30961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:38.361464 30961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:51:38.361491 30961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:38.367574 30961 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.131:39589
I20260928 22:51:38.367651 31089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.131:39589 every 8 connection(s)
I20260928 22:51:38.368005 30961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
Sep 28 22:51:38 dist-test-slave-56vw krb5kdc[30586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635898, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.131@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:51:38.374918 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 30961
I20260928 22:51:38.374996 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance
I20260928 22:51:38.375190 16870 external_mini_cluster.cc:1428] Setting key 59317c992e63c93719559ba58d8adeac
I20260928 22:51:38.378201 30680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:38.368793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:54191 (local address 127.16.121.190:37803)
0928 22:51:38.368860 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:38.368861 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:38.369125 (+ 264us) server_negotiation.cc:407] Connection header received
0928 22:51:38.369622 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:38.369623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:38.369636 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:38.369713 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:51:38.370830 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:38.371461 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:38.372503 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:38.372715 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:38.372782 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:38.375163 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:51:38.375164 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:51:38.375165 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:51:38.375181 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:51:38.376763 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:38.377187 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:38.377188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:38.377189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:38.377246 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:38.377527 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:38.377528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:38.377529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:38.377661 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:51:38.377731 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:38.378020 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 22:51:38.378102 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 22:51:38.378422 31092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:38.368766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.131:54191)
0928 22:51:38.368958 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:38.369002 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 22:51:38.369509 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:51:38.369796 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:51:38.369800 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:51:38.370423 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:51:38.370650 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:38.370656 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:38.371555 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:38.371556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:38.372395 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:38.372397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:38.372803 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:38.372804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:38.372909 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:38.373500 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:51:38.373509 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:51:38.375006 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:51:38.376862 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:38.376869 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:38.376871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:38.377105 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:38.377327 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:38.377328 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:38.377328 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:38.377451 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:38.377734 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:51:38.377741 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:51:38.377886 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 22:51:38.377998 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 22:51:38.378170 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":113,"threads_started":1}
I20260928 22:51:38.379101 31090 heartbeater.cc:347] Connected to a master server at 127.16.121.190:37803
I20260928 22:51:38.379200 31090 heartbeater.cc:464] Registering TS with master...
I20260928 22:51:38.379407 31090 heartbeater.cc:511] Master 127.16.121.190:37803 requested a full tablet report, sending...
I20260928 22:51:38.379921 30631 ts_manager.cc:194] Registered new tserver with Master: 87072ae366cd40b48beee55f28afb7b9 (127.16.121.131:39589)
I20260928 22:51:38.380345 30631 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.131@KRBTEST.COM'} at 127.16.121.131:54191
I20260928 22:51:38.387444 16870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:51:38.395205 30680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:38.388964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59880 (local address 127.16.121.190:37803)
0928 22:51:38.389022 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:38.389024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:38.389134 (+ 110us) server_negotiation.cc:407] Connection header received
0928 22:51:38.389182 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:38.389183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:38.389191 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:38.389244 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:51:38.389891 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:38.390445 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:38.391161 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:38.391342 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:38.391417 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:38.392564 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:51:38.392565 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:51:38.392565 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:51:38.392572 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:51:38.393898 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:38.394277 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:38.394278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:38.394278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:38.394338 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:38.394520 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:38.394521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:38.394521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:38.394668 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:51:38.394750 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:38.394991 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 22:51:38.395129 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 22:51:38.396931 30631 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59880:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:51:38.398514 30631 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:51:38.406198 31105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:38.401723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58920 (local address 127.16.121.129:36367)
0928 22:51:38.402121 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:38.402127 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:38.402138 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:51:38.402193 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:38.402196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:38.402214 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:38.402296 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:51:38.402562 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:38.403686 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:38.405001 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:38.405700 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:38.405820 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:38.405889 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:38.406054 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 22:51:38.406098 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":96,"threads_started":1}
I20260928 22:51:38.406188 31101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:38.401463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:36367 (local address 127.0.0.1:58920)
0928 22:51:38.401911 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:38.401923 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:51:38.401971 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:51:38.402388 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:51:38.402392 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:51:38.402394 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:51:38.402470 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:38.402475 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:38.403795 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:38.403796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:38.404890 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:38.404913 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:38.405821 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:38.405822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:38.405912 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:38.405928 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 22:51:38.406036 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 22:51:38.406098 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"mutex_wait_us":73,"thread_start_us":98,"threads_started":1}
I20260928 22:51:38.406868 31102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:38.401438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.130:41643 (local address 127.0.0.1:57928)
0928 22:51:38.401715 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:38.401796 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0928 22:51:38.401904 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:51:38.402388 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:51:38.402392 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:51:38.402421 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:51:38.402470 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:38.402472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:38.403969 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:38.403971 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:38.405237 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:38.405240 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:38.406231 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:38.406232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:38.406668 (+ 436us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:38.406673 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:51:38.406753 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 22:51:38.406792 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":60,"threads_started":1}
I20260928 22:51:38.406971 31104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:38.401684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57928 (local address 127.16.121.130:41643)
0928 22:51:38.402230 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:38.402235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:38.402245 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:51:38.402284 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:38.402286 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:38.402302 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:38.402383 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:51:38.402609 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:38.403846 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:38.405339 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:38.406075 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:38.406140 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:38.406197 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:38.406825 (+ 628us) server_negotiation.cc:299] Negotiation successful
0928 22:51:38.406875 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":114,"threads_started":1}
I20260928 22:51:38.407135 30756 tablet_service.cc:1467] Processing CreateTablet for tablet df8170eee726457ea285c53bc5b909ae (DEFAULT_TABLE table=test-table [id=0bd344fa86a2463990916adcc2eabf0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:51:38.407600 30756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df8170eee726457ea285c53bc5b909ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:51:38.407658 31103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:38.401511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.131:39589 (local address 127.0.0.1:55964)
0928 22:51:38.402168 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:38.402180 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:51:38.402251 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:51:38.403428 (+ 1177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:51:38.403430 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:51:38.403432 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:51:38.403481 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:38.403482 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:38.404986 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:38.404987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:38.406020 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:38.406024 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:38.407414 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:38.407415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:38.407481 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:38.407483 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 22:51:38.407552 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 22:51:38.407593 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":99,"threads_started":1}
I20260928 22:51:38.407881 31106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:38.402896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55964 (local address 127.16.121.131:39589)
0928 22:51:38.403209 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:38.403213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:38.403225 (+ 12us) server_negotiation.cc:407] Connection header received
0928 22:51:38.403291 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:38.403293 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:38.403310 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:38.403410 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:51:38.403549 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:38.404818 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:38.406443 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:38.407238 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:38.407295 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:38.407352 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:38.407569 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 22:51:38.407612 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":77,"threads_started":1}
I20260928 22:51:38.407964 30890 tablet_service.cc:1467] Processing CreateTablet for tablet df8170eee726457ea285c53bc5b909ae (DEFAULT_TABLE table=test-table [id=0bd344fa86a2463990916adcc2eabf0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:51:38.408337 30890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df8170eee726457ea285c53bc5b909ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:51:38.411636 31107 tablet_bootstrap.cc:492] T df8170eee726457ea285c53bc5b909ae P 86d13097cb5348d2a58f4c202689f82e: Bootstrap starting.
I20260928 22:51:38.411772 31024 tablet_service.cc:1467] Processing CreateTablet for tablet df8170eee726457ea285c53bc5b909ae (DEFAULT_TABLE table=test-table [id=0bd344fa86a2463990916adcc2eabf0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:51:38.412004 31108 tablet_bootstrap.cc:492] T df8170eee726457ea285c53bc5b909ae P 90486aa702d140f19e71be00c5088990: Bootstrap starting.
I20260928 22:51:38.412133 31024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df8170eee726457ea285c53bc5b909ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:51:38.412390 31107 tablet_bootstrap.cc:654] T df8170eee726457ea285c53bc5b909ae P 86d13097cb5348d2a58f4c202689f82e: Neither blocks nor log segments found. Creating new log.
I20260928 22:51:38.412662 31108 tablet_bootstrap.cc:654] T df8170eee726457ea285c53bc5b909ae P 90486aa702d140f19e71be00c5088990: Neither blocks nor log segments found. Creating new log.
I20260928 22:51:38.412874 31107 log.cc:824] T df8170eee726457ea285c53bc5b909ae P 86d13097cb5348d2a58f4c202689f82e: Log is configured to *not* fsync() on all Append() calls
I20260928 22:51:38.413213 31108 log.cc:824] T df8170eee726457ea285c53bc5b909ae P 90486aa702d140f19e71be00c5088990: Log is configured to *not* fsync() on all Append() calls
I20260928 22:51:38.413683 31107 tablet_bootstrap.cc:492] T df8170eee726457ea285c53bc5b909ae P 86d13097cb5348d2a58f4c202689f82e: No bootstrap required, opened a new log
I20260928 22:51:38.413753 31107 ts_tablet_manager.cc:1397] T df8170eee726457ea285c53bc5b909ae P 86d13097cb5348d2a58f4c202689f82e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:51:38.413997 31108 tablet_bootstrap.cc:492] T df8170eee726457ea285c53bc5b909ae P 90486aa702d140f19e71be00c5088990: No bootstrap required, opened a new log
I20260928 22:51:38.414057 31108 ts_tablet_manager.cc:1397] T df8170eee726457ea285c53bc5b909ae P 90486aa702d140f19e71be00c5088990: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:51:38.415369 31107 raft_consensus.cc:348] T df8170eee726457ea285c53bc5b909ae P 86d13097cb5348d2a58f4c202689f82e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90486aa702d140f19e71be00c5088990" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 41643 } } peers { permanent_uuid: "87072ae366cd40b48beee55f28afb7b9" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 39589 } } peers { permanent_uuid: "86d13097cb5348d2a58f4c202689f82e" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 36367 } }
I20260928 22:51:38.415571 31107 raft_consensus.cc:374] T df8170eee726457ea285c53bc5b909ae P 86d13097cb5348d2a58f4c202689f82e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:51:38.415621 31107 raft_consensus.cc:729] T df8170eee726457ea285c53bc5b909ae P 86d13097cb5348d2a58f4c202689f82e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86d13097cb5348d2a58f4c202689f82e, State: Initialized, Role: FOLLOWER
I20260928 22:51:38.415748 31107 consensus_queue.cc:260] T df8170eee726457ea285c53bc5b909ae P 86d13097cb5348d2a58f4c202689f82e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90486aa702d140f19e71be00c5088990" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 41643 } } peers { permanent_uuid: "87072ae366cd40b48beee55f28afb7b9" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 39589 } } peers { permanent_uuid: "86d13097cb5348d2a58f4c202689f82e" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 36367 } }
I20260928 22:51:38.415794 31111 tablet_bootstrap.cc:492] T df8170eee726457ea285c53bc5b909ae P 87072ae366cd40b48beee55f28afb7b9: Bootstrap starting.
I20260928 22:51:38.416079 31107 ts_tablet_manager.cc:1428] T df8170eee726457ea285c53bc5b909ae P 86d13097cb5348d2a58f4c202689f82e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 22:51:38.416245 30822 heartbeater.cc:503] Master 127.16.121.190:37803 was elected leader, sending a full tablet report...
I20260928 22:51:38.416472 31111 tablet_bootstrap.cc:654] T df8170eee726457ea285c53bc5b909ae P 87072ae366cd40b48beee55f28afb7b9: Neither blocks nor log segments found. Creating new log.
I20260928 22:51:38.416543 31108 raft_consensus.cc:348] T df8170eee726457ea285c53bc5b909ae P 90486aa702d140f19e71be00c5088990 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90486aa702d140f19e71be00c5088990" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 41643 } } peers { permanent_uuid: "87072ae366cd40b48beee55f28afb7b9" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 39589 } } peers { permanent_uuid: "86d13097cb5348d2a58f4c202689f82e" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 36367 } }
I20260928 22:51:38.416731 31108 raft_consensus.cc:374] T df8170eee726457ea285c53bc5b909ae P 90486aa702d140f19e71be00c5088990 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:51:38.416770 31108 raft_consensus.cc:729] T df8170eee726457ea285c53bc5b909ae P 90486aa702d140f19e71be00c5088990 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90486aa702d140f19e71be00c5088990, State: Initialized, Role: FOLLOWER
I20260928 22:51:38.416929 31108 consensus_queue.cc:260] T df8170eee726457ea285c53bc5b909ae P 90486aa702d140f19e71be00c5088990 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90486aa702d140f19e71be00c5088990" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 41643 } } peers { permanent_uuid: "87072ae366cd40b48beee55f28afb7b9" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 39589 } } peers { permanent_uuid: "86d13097cb5348d2a58f4c202689f82e" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 36367 } }
I20260928 22:51:38.417023 31111 log.cc:824] T df8170eee726457ea285c53bc5b909ae P 87072ae366cd40b48beee55f28afb7b9: Log is configured to *not* fsync() on all Append() calls
I20260928 22:51:38.417238 31108 ts_tablet_manager.cc:1428] T df8170eee726457ea285c53bc5b909ae P 90486aa702d140f19e71be00c5088990: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:51:38.417349 30956 heartbeater.cc:503] Master 127.16.121.190:37803 was elected leader, sending a full tablet report...
I20260928 22:51:38.417933 31111 tablet_bootstrap.cc:492] T df8170eee726457ea285c53bc5b909ae P 87072ae366cd40b48beee55f28afb7b9: No bootstrap required, opened a new log
I20260928 22:51:38.418007 31111 ts_tablet_manager.cc:1397] T df8170eee726457ea285c53bc5b909ae P 87072ae366cd40b48beee55f28afb7b9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:51:38.419747 31111 raft_consensus.cc:348] T df8170eee726457ea285c53bc5b909ae P 87072ae366cd40b48beee55f28afb7b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90486aa702d140f19e71be00c5088990" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 41643 } } peers { permanent_uuid: "87072ae366cd40b48beee55f28afb7b9" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 39589 } } peers { permanent_uuid: "86d13097cb5348d2a58f4c202689f82e" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 36367 } }
I20260928 22:51:38.419929 31111 raft_consensus.cc:374] T df8170eee726457ea285c53bc5b909ae P 87072ae366cd40b48beee55f28afb7b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:51:38.419960 31111 raft_consensus.cc:729] T df8170eee726457ea285c53bc5b909ae P 87072ae366cd40b48beee55f28afb7b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87072ae366cd40b48beee55f28afb7b9, State: Initialized, Role: FOLLOWER
I20260928 22:51:38.420099 31111 consensus_queue.cc:260] T df8170eee726457ea285c53bc5b909ae P 87072ae366cd40b48beee55f28afb7b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90486aa702d140f19e71be00c5088990" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 41643 } } peers { permanent_uuid: "87072ae366cd40b48beee55f28afb7b9" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 39589 } } peers { permanent_uuid: "86d13097cb5348d2a58f4c202689f82e" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 36367 } }
I20260928 22:51:38.420408 31111 ts_tablet_manager.cc:1428] T df8170eee726457ea285c53bc5b909ae P 87072ae366cd40b48beee55f28afb7b9: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 22:51:38.420430 31090 heartbeater.cc:503] Master 127.16.121.190:37803 was elected leader, sending a full tablet report...
W20260928 22:51:38.455897 30957 tablet.cc:2369] T df8170eee726457ea285c53bc5b909ae P 90486aa702d140f19e71be00c5088990: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:51:38.521756 30823 tablet.cc:2369] T df8170eee726457ea285c53bc5b909ae P 86d13097cb5348d2a58f4c202689f82e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:51:38.529213 31112 raft_consensus.cc:482] T df8170eee726457ea285c53bc5b909ae P 86d13097cb5348d2a58f4c202689f82e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:51:38.529335 31112 raft_consensus.cc:504] T df8170eee726457ea285c53bc5b909ae P 86d13097cb5348d2a58f4c202689f82e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90486aa702d140f19e71be00c5088990" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 41643 } } peers { permanent_uuid: "87072ae366cd40b48beee55f28afb7b9" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 39589 } } peers { permanent_uuid: "86d13097cb5348d2a58f4c202689f82e" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 36367 } }
I20260928 22:51:38.529656 31112 leader_election.cc:288] T df8170eee726457ea285c53bc5b909ae P 86d13097cb5348d2a58f4c202689f82e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90486aa702d140f19e71be00c5088990 (127.16.121.130:41643), 87072ae366cd40b48beee55f28afb7b9 (127.16.121.131:39589)
I20260928 22:51:38.533804 31106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:38.529983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:44631 (local address 127.16.121.131:39589)
0928 22:51:38.530036 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:38.530037 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:38.530046 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:51:38.530082 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:38.530082 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:38.530090 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:38.530186 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:51:38.530447 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:38.531245 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:38.532877 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:38.533437 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:38.533514 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:38.533522 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:38.533690 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 22:51:38.533728 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 22:51:38.533804 31104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:38.529968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:41367 (local address 127.16.121.130:41643)
0928 22:51:38.530015 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:38.530017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:38.530150 (+ 133us) server_negotiation.cc:407] Connection header received
0928 22:51:38.530204 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:38.530205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:38.530215 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:38.530275 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:51:38.530452 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:38.531240 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:38.532819 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:38.533422 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:38.533509 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:38.533519 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:38.533689 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 22:51:38.533728 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 22:51:38.533843 30824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:38.529885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.131:39589 (local address 127.16.121.129:44631)
0928 22:51:38.529928 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:38.529942 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 22:51:38.529998 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:51:38.530208 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:51:38.530209 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:51:38.530211 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:51:38.530310 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:38.530312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:38.531341 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:38.531343 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:38.532707 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:38.532710 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:38.533520 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:38.533521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:38.533608 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:38.533609 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 22:51:38.533669 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 22:51:38.533727 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":66}
I20260928 22:51:38.533850 31116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:38.529886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.130:41643 (local address 127.16.121.129:41367)
0928 22:51:38.530077 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:38.530086 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:51:38.530134 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:51:38.530290 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:51:38.530291 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:51:38.530292 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:51:38.530357 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:38.530359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:38.531351 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:38.531351 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:38.532707 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:38.532710 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:38.533520 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:38.533521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:38.533604 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:38.533609 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:51:38.533669 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 22:51:38.533727 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":148,"thread_start_us":82,"threads_started":1}
I20260928 22:51:38.534265 31044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8170eee726457ea285c53bc5b909ae" candidate_uuid: "86d13097cb5348d2a58f4c202689f82e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87072ae366cd40b48beee55f28afb7b9" is_pre_election: true
I20260928 22:51:38.534435 31044 raft_consensus.cc:2454] T df8170eee726457ea285c53bc5b909ae P 87072ae366cd40b48beee55f28afb7b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86d13097cb5348d2a58f4c202689f82e in term 0.
I20260928 22:51:38.534456 30910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8170eee726457ea285c53bc5b909ae" candidate_uuid: "86d13097cb5348d2a58f4c202689f82e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90486aa702d140f19e71be00c5088990" is_pre_election: true
I20260928 22:51:38.534624 30910 raft_consensus.cc:2454] T df8170eee726457ea285c53bc5b909ae P 90486aa702d140f19e71be00c5088990 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86d13097cb5348d2a58f4c202689f82e in term 0.
I20260928 22:51:38.534740 30711 leader_election.cc:302] T df8170eee726457ea285c53bc5b909ae P 86d13097cb5348d2a58f4c202689f82e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86d13097cb5348d2a58f4c202689f82e, 87072ae366cd40b48beee55f28afb7b9; no voters:
I20260928 22:51:38.534929 31112 raft_consensus.cc:2781] T df8170eee726457ea285c53bc5b909ae P 86d13097cb5348d2a58f4c202689f82e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:51:38.535004 31112 raft_consensus.cc:482] T df8170eee726457ea285c53bc5b909ae P 86d13097cb5348d2a58f4c202689f82e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:51:38.535038 31112 raft_consensus.cc:3037] T df8170eee726457ea285c53bc5b909ae P 86d13097cb5348d2a58f4c202689f82e [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:51:38.535786 31112 raft_consensus.cc:504] T df8170eee726457ea285c53bc5b909ae P 86d13097cb5348d2a58f4c202689f82e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90486aa702d140f19e71be00c5088990" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 41643 } } peers { permanent_uuid: "87072ae366cd40b48beee55f28afb7b9" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 39589 } } peers { permanent_uuid: "86d13097cb5348d2a58f4c202689f82e" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 36367 } }
I20260928 22:51:38.535946 31112 leader_election.cc:288] T df8170eee726457ea285c53bc5b909ae P 86d13097cb5348d2a58f4c202689f82e [CANDIDATE]: Term 1 election: Requested vote from peers 90486aa702d140f19e71be00c5088990 (127.16.121.130:41643), 87072ae366cd40b48beee55f28afb7b9 (127.16.121.131:39589)
I20260928 22:51:38.536098 31044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8170eee726457ea285c53bc5b909ae" candidate_uuid: "86d13097cb5348d2a58f4c202689f82e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87072ae366cd40b48beee55f28afb7b9"
I20260928 22:51:38.536109 30910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8170eee726457ea285c53bc5b909ae" candidate_uuid: "86d13097cb5348d2a58f4c202689f82e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90486aa702d140f19e71be00c5088990"
I20260928 22:51:38.536183 30910 raft_consensus.cc:3037] T df8170eee726457ea285c53bc5b909ae P 90486aa702d140f19e71be00c5088990 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:51:38.536206 31044 raft_consensus.cc:3037] T df8170eee726457ea285c53bc5b909ae P 87072ae366cd40b48beee55f28afb7b9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:51:38.536943 30910 raft_consensus.cc:2454] T df8170eee726457ea285c53bc5b909ae P 90486aa702d140f19e71be00c5088990 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86d13097cb5348d2a58f4c202689f82e in term 1.
I20260928 22:51:38.536952 31044 raft_consensus.cc:2454] T df8170eee726457ea285c53bc5b909ae P 87072ae366cd40b48beee55f28afb7b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86d13097cb5348d2a58f4c202689f82e in term 1.
I20260928 22:51:38.537119 30710 leader_election.cc:302] T df8170eee726457ea285c53bc5b909ae P 86d13097cb5348d2a58f4c202689f82e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86d13097cb5348d2a58f4c202689f82e, 90486aa702d140f19e71be00c5088990; no voters:
I20260928 22:51:38.537268 31112 raft_consensus.cc:2781] T df8170eee726457ea285c53bc5b909ae P 86d13097cb5348d2a58f4c202689f82e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:51:38.537487 31112 raft_consensus.cc:686] T df8170eee726457ea285c53bc5b909ae P 86d13097cb5348d2a58f4c202689f82e [term 1 LEADER]: Becoming Leader. State: Replica: 86d13097cb5348d2a58f4c202689f82e, State: Running, Role: LEADER
I20260928 22:51:38.537567 31112 consensus_queue.cc:237] T df8170eee726457ea285c53bc5b909ae P 86d13097cb5348d2a58f4c202689f82e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90486aa702d140f19e71be00c5088990" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 41643 } } peers { permanent_uuid: "87072ae366cd40b48beee55f28afb7b9" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 39589 } } peers { permanent_uuid: "86d13097cb5348d2a58f4c202689f82e" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 36367 } }
I20260928 22:51:38.538625 30631 catalog_manager.cc:5495] T df8170eee726457ea285c53bc5b909ae P 86d13097cb5348d2a58f4c202689f82e reported cstate change: term changed from 0 to 1, leader changed from <none> to 86d13097cb5348d2a58f4c202689f82e (127.16.121.129). New cstate: current_term: 1 leader_uuid: "86d13097cb5348d2a58f4c202689f82e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90486aa702d140f19e71be00c5088990" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 41643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87072ae366cd40b48beee55f28afb7b9" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 39589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86d13097cb5348d2a58f4c202689f82e" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 36367 } health_report { overall_health: HEALTHY } } }
I20260928 22:51:38.554032 31105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:38.551480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58930 (local address 127.16.121.129:36367)
0928 22:51:38.551539 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:38.551541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:38.551550 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:51:38.551614 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:38.551615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:38.551633 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:38.551698 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:51:38.551840 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:38.552326 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:38.553141 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:38.553378 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:38.553455 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:38.553593 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:51:38.553723 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:51:38.553800 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:38.553869 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 22:51:38.553888 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 22:51:38.555899 30910 raft_consensus.cc:1261] T df8170eee726457ea285c53bc5b909ae P 90486aa702d140f19e71be00c5088990 [term 1 FOLLOWER]: Refusing update from remote peer 86d13097cb5348d2a58f4c202689f82e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:51:38.555940 31044 raft_consensus.cc:1261] T df8170eee726457ea285c53bc5b909ae P 87072ae366cd40b48beee55f28afb7b9 [term 1 FOLLOWER]: Refusing update from remote peer 86d13097cb5348d2a58f4c202689f82e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:51:38.556227 31112 consensus_queue.cc:1035] T df8170eee726457ea285c53bc5b909ae P 86d13097cb5348d2a58f4c202689f82e [LEADER]: Connected to new peer: Peer: permanent_uuid: "90486aa702d140f19e71be00c5088990" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 41643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:51:38.556406 31117 consensus_queue.cc:1035] T df8170eee726457ea285c53bc5b909ae P 86d13097cb5348d2a58f4c202689f82e [LEADER]: Connected to new peer: Peer: permanent_uuid: "87072ae366cd40b48beee55f28afb7b9" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 39589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:51:38.558776 31120 mvcc.cc:205] Tried to move back new op lower bound from 7334444640482201600 to 7334444640410734592. Current Snapshot: MvccSnapshot[applied={T|T < 7334444640482201600}]
I20260928 22:51:38.558964 31121 mvcc.cc:205] Tried to move back new op lower bound from 7334444640482201600 to 7334444640410734592. Current Snapshot: MvccSnapshot[applied={T|T < 7334444640482201600}]
I20260928 22:51:38.559381 31122 mvcc.cc:205] Tried to move back new op lower bound from 7334444640482201600 to 7334444640410734592. Current Snapshot: MvccSnapshot[applied={T|T < 7334444640482201600}]
I20260928 22:51:38.561035 30630 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:51:38.561143 30630 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:51:38.561941 30630 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447: Sending DeleteTablet for 3 replicas of tablet df8170eee726457ea285c53bc5b909ae
I20260928 22:51:38.562287 30756 tablet_service.cc:1514] Processing DeleteTablet for tablet df8170eee726457ea285c53bc5b909ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:51:38 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:58920
I20260928 22:51:38.562296 31024 tablet_service.cc:1514] Processing DeleteTablet for tablet df8170eee726457ea285c53bc5b909ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:51:38 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:55964
I20260928 22:51:38.562319 30890 tablet_service.cc:1514] Processing DeleteTablet for tablet df8170eee726457ea285c53bc5b909ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:51:38 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:57928
I20260928 22:51:38.562537 31130 tablet_replica.cc:330] T df8170eee726457ea285c53bc5b909ae P 87072ae366cd40b48beee55f28afb7b9: stopping tablet replica
I20260928 22:51:38.562569 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 30693
I20260928 22:51:38.563153 31130 raft_consensus.cc:2229] T df8170eee726457ea285c53bc5b909ae P 87072ae366cd40b48beee55f28afb7b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:51:38.563298 31130 raft_consensus.cc:2258] T df8170eee726457ea285c53bc5b909ae P 87072ae366cd40b48beee55f28afb7b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:51:38.563668 31130 ts_tablet_manager.cc:1905] T df8170eee726457ea285c53bc5b909ae P 87072ae366cd40b48beee55f28afb7b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:51:38.565558 31130 ts_tablet_manager.cc:1918] T df8170eee726457ea285c53bc5b909ae P 87072ae366cd40b48beee55f28afb7b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:51:38.565663 31130 log.cc:1196] T df8170eee726457ea285c53bc5b909ae P 87072ae366cd40b48beee55f28afb7b9: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-2/wal/wals/df8170eee726457ea285c53bc5b909ae
I20260928 22:51:38.565929 31130 ts_tablet_manager.cc:1939] T df8170eee726457ea285c53bc5b909ae P 87072ae366cd40b48beee55f28afb7b9: Deleting consensus metadata
I20260928 22:51:38.566393 30617 catalog_manager.cc:4841] TS 87072ae366cd40b48beee55f28afb7b9 (127.16.121.131:39589): tablet df8170eee726457ea285c53bc5b909ae (table test-table [id=0bd344fa86a2463990916adcc2eabf0b]) successfully deleted
I20260928 22:51:38.567191 31131 tablet_replica.cc:330] T df8170eee726457ea285c53bc5b909ae P 90486aa702d140f19e71be00c5088990: stopping tablet replica
I20260928 22:51:38.567332 31131 raft_consensus.cc:2229] T df8170eee726457ea285c53bc5b909ae P 90486aa702d140f19e71be00c5088990 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:51:38.567477 31131 raft_consensus.cc:2258] T df8170eee726457ea285c53bc5b909ae P 90486aa702d140f19e71be00c5088990 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:51:38.567906 31131 ts_tablet_manager.cc:1905] T df8170eee726457ea285c53bc5b909ae P 90486aa702d140f19e71be00c5088990: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 22:51:38.569165 30618 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.121.129:36367 (error 108)
I20260928 22:51:38.569309 31131 ts_tablet_manager.cc:1918] T df8170eee726457ea285c53bc5b909ae P 90486aa702d140f19e71be00c5088990: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:51:38.569386 31131 log.cc:1196] T df8170eee726457ea285c53bc5b909ae P 90486aa702d140f19e71be00c5088990: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-1/wal/wals/df8170eee726457ea285c53bc5b909ae
I20260928 22:51:38.569429 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 30827
I20260928 22:51:38.570274 31103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:38.570131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:36367 (local address 127.0.0.1:58944)
0928 22:51:38.570196 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:38.570235 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.129:36367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 22:51:38.570392 30618 catalog_manager.cc:4584] TS 86d13097cb5348d2a58f4c202689f82e (127.16.121.129:36367): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet df8170eee726457ea285c53bc5b909ae: Network error: Client connection negotiation failed: client connection to 127.16.121.129:36367: connect: Connection refused (error 111)
I20260928 22:51:38.574870 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 30961
I20260928 22:51:38.574906 31103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:38.574773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.130:41643 (local address 127.0.0.1:57932)
0928 22:51:38.574843 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:38.574877 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.130:41643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 22:51:38.580274 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 30599
I20260928 22:51:38.585687 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:37803
--webserver_interface=127.16.121.190
--webserver_port=39081
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:36021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:37803
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:51:38.668373 31134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:51:38.668642 31134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:51:38.668681 31134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:51:38.669873 31134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:51:38.669945 31134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:51:38.669966 31134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:51:38.669987 31134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:51:38.670017 31134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:51:38.671540 31134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:36021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:37803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:37803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.121.190
--webserver_port=39081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:51:38.671828 31134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:51:38.672070 31134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:51:38.681107 31143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:51:38.681248 31134 server_base.cc:1034] running on GCE node
W20260928 22:51:38.681125 31140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:51:38.681404 31141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:51:38.681665 31134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:51:38.682011 31134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:51:38.683179 31134 hybrid_clock.cc:648] HybridClock initialized: now 1790635898683168 us; error 43 us; skew 500 ppm
Sep 28 22:51:38 dist-test-slave-56vw krb5kdc[30586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635898, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:51:38.685806 31134 init.cc:377] Logged in from keytab as kudu/127.16.121.190@KRBTEST.COM (short username kudu)
I20260928 22:51:38.686327 31134 webserver.cc:460] Webserver started at http://127.16.121.190:39081/ using document root <none> and password file <none>
I20260928 22:51:38.686525 31134 fs_manager.cc:359] Metadata directory not provided
I20260928 22:51:38.686576 31134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:51:38.688036 31134 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:51:38.688727 31150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:38.688933 31134 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:51:38.689028 31134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "2fd24b851da0411193bead83d7411447"
format_stamp: "Formatted at 2026-09-28 22:51:37 on dist-test-slave-56vw"
server_key: "93823709de34f9f37ec9d781eb76d7b7"
server_key_iv: "a281b7f64ec9c6a39a0ca05d6f29e18b"
server_key_version: "encryptionkey@0"
I20260928 22:51:38.689435 31134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:51:38.696606 31134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:51:38.698503 31134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:51:38.698614 31134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:51:38.703176 31134 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.190:37803
I20260928 22:51:38.703213 31202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.190:37803 every 8 connection(s)
I20260928 22:51:38.703644 31134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:51:38.705273 31203 sys_catalog.cc:263] Verifying existing consensus state
I20260928 22:51:38.705955 31203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447: Bootstrap starting.
I20260928 22:51:38.711170 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 31134
I20260928 22:51:38.711478 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.129:36367
--local_ip_for_outbound_sockets=127.16.121.129
--tserver_master_addrs=127.16.121.190:37803
--webserver_port=46821
--webserver_interface=127.16.121.129
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab
--principal=kudu/127.16.121.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:36021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:51:38.714537 31203 log.cc:824] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447: Log is configured to *not* fsync() on all Append() calls
I20260928 22:51:38.716622 31203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 22:51:38.716938 31203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447: Bootstrap complete.
I20260928 22:51:38.719363 31203 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fd24b851da0411193bead83d7411447" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37803 } }
I20260928 22:51:38.719631 31203 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fd24b851da0411193bead83d7411447, State: Initialized, Role: FOLLOWER
I20260928 22:51:38.719825 31203 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fd24b851da0411193bead83d7411447" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37803 } }
I20260928 22:51:38.719906 31203 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:51:38.719967 31203 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:51:38.720018 31203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [term 1 FOLLOWER]: Advancing to term 2
I20260928 22:51:38.721093 31203 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fd24b851da0411193bead83d7411447" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37803 } }
I20260928 22:51:38.721228 31203 leader_election.cc:302] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fd24b851da0411193bead83d7411447; no voters:
I20260928 22:51:38.721447 31203 leader_election.cc:288] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 22:51:38.721487 31207 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 22:51:38.721602 31207 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [term 2 LEADER]: Becoming Leader. State: Replica: 2fd24b851da0411193bead83d7411447, State: Running, Role: LEADER
I20260928 22:51:38.721700 31207 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fd24b851da0411193bead83d7411447" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37803 } }
I20260928 22:51:38.721751 31203 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:51:38.721979 31208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2fd24b851da0411193bead83d7411447" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fd24b851da0411193bead83d7411447" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37803 } } }
I20260928 22:51:38.722074 31208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [sys.catalog]: This master's current role is: LEADER
I20260928 22:51:38.721974 31209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fd24b851da0411193bead83d7411447. Latest consensus state: current_term: 2 leader_uuid: "2fd24b851da0411193bead83d7411447" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fd24b851da0411193bead83d7411447" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37803 } } }
I20260928 22:51:38.722254 31209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [sys.catalog]: This master's current role is: LEADER
I20260928 22:51:38.722409 31211 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:51:38.723301 31211 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:51:38.723435 31211 catalog_manager.cc:1259] Loaded cluster ID: a02d1fff1901438a888b8cd9d3c64119
I20260928 22:51:38.723466 31211 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:51:38.723726 31211 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 22:51:38.728933 31211 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 22:51:38.732565 31211 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:51:38.732831 31211 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447: Loaded TSK: 0
I20260928 22:51:38.733083 31211 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 22:51:38.798870 31205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:51:38.799095 31205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:51:38.799151 31205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:51:38.800344 31205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:51:38.800384 31205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:51:38.800411 31205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.129
I20260928 22:51:38.801934 31205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:36021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab
--principal=kudu/127.16.121.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.129:36367
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.121.129
--webserver_port=46821
--tserver_master_addrs=127.16.121.190:37803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.129
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:51:38.802174 31205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:51:38.802405 31205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:51:38.808616 31231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:51:38.808670 31233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:51:38.808635 31230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:51:38.808905 31205 server_base.cc:1034] running on GCE node
I20260928 22:51:38.809149 31205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:51:38.809466 31205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:51:38.810648 31205 hybrid_clock.cc:648] HybridClock initialized: now 1790635898810640 us; error 38 us; skew 500 ppm
Sep 28 22:51:38 dist-test-slave-56vw krb5kdc[30586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635898, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:51:38.813755 31205 init.cc:377] Logged in from keytab as kudu/127.16.121.129@KRBTEST.COM (short username kudu)
I20260928 22:51:38.814302 31205 webserver.cc:460] Webserver started at http://127.16.121.129:46821/ using document root <none> and password file <none>
I20260928 22:51:38.814574 31205 fs_manager.cc:359] Metadata directory not provided
I20260928 22:51:38.814656 31205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:51:38.816612 31205 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:51:38.817432 31240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:38.817780 31205 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:51:38.817875 31205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-0/wal
uuid: "86d13097cb5348d2a58f4c202689f82e"
format_stamp: "Formatted at 2026-09-28 22:51:37 on dist-test-slave-56vw"
server_key: "d1ab9569fc2b4e447e0579f94ac0eb03"
server_key_iv: "06a6de36040dba02659a2d0aa34e60a8"
server_key_version: "encryptionkey@0"
I20260928 22:51:38.818336 31205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:51:38.838136 31205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:51:38.840852 31205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:51:38.841027 31205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:51:38.841300 31205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:51:38.841838 31247 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 22:51:38.843446 31205 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 22:51:38.843518 31205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 22:51:38.843554 31205 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 22:51:38.844408 31205 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 22:51:38.844463 31205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 22:51:38.844561 31247 tablet_bootstrap.cc:492] T df8170eee726457ea285c53bc5b909ae P 86d13097cb5348d2a58f4c202689f82e: Bootstrap starting.
I20260928 22:51:38.851372 31205 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.129:36367
I20260928 22:51:38.851740 31354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.129:36367 every 8 connection(s)
I20260928 22:51:38.851994 31205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
I20260928 22:51:38.856698 31247 log.cc:824] T df8170eee726457ea285c53bc5b909ae P 86d13097cb5348d2a58f4c202689f82e: Log is configured to *not* fsync() on all Append() calls
I20260928 22:51:38.857064 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 31205
I20260928 22:51:38.857420 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.130:41643
--local_ip_for_outbound_sockets=127.16.121.130
--tserver_master_addrs=127.16.121.190:37803
--webserver_port=42105
--webserver_interface=127.16.121.130
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab
--principal=kudu/127.16.121.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:36021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:51:38.859848 31247 tablet_bootstrap.cc:492] T df8170eee726457ea285c53bc5b909ae P 86d13097cb5348d2a58f4c202689f82e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 22:51:38.860188 31247 tablet_bootstrap.cc:492] T df8170eee726457ea285c53bc5b909ae P 86d13097cb5348d2a58f4c202689f82e: Bootstrap complete.
I20260928 22:51:38.861533 31247 ts_tablet_manager.cc:1397] T df8170eee726457ea285c53bc5b909ae P 86d13097cb5348d2a58f4c202689f82e: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.011s
I20260928 22:51:38.863468 31247 raft_consensus.cc:348] T df8170eee726457ea285c53bc5b909ae P 86d13097cb5348d2a58f4c202689f82e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90486aa702d140f19e71be00c5088990" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 41643 } } peers { permanent_uuid: "87072ae366cd40b48beee55f28afb7b9" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 39589 } } peers { permanent_uuid: "86d13097cb5348d2a58f4c202689f82e" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 36367 } }
I20260928 22:51:38.863865 31247 raft_consensus.cc:729] T df8170eee726457ea285c53bc5b909ae P 86d13097cb5348d2a58f4c202689f82e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86d13097cb5348d2a58f4c202689f82e, State: Initialized, Role: FOLLOWER
I20260928 22:51:38.864043 31247 consensus_queue.cc:260] T df8170eee726457ea285c53bc5b909ae P 86d13097cb5348d2a58f4c202689f82e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90486aa702d140f19e71be00c5088990" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 41643 } } peers { permanent_uuid: "87072ae366cd40b48beee55f28afb7b9" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 39589 } } peers { permanent_uuid: "86d13097cb5348d2a58f4c202689f82e" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 36367 } }
I20260928 22:51:38.864326 31247 ts_tablet_manager.cc:1428] T df8170eee726457ea285c53bc5b909ae P 86d13097cb5348d2a58f4c202689f82e: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
Sep 28 22:51:38 dist-test-slave-56vw krb5kdc[30586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635898, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:51:38.869052 31358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:38.852865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:56527 (local address 127.16.121.190:37803)
0928 22:51:38.854339 (+ 1474us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:38.854350 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:38.854406 (+ 56us) server_negotiation.cc:407] Connection header received
0928 22:51:38.854579 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:38.854592 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:38.854675 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:38.854868 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:51:38.855771 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:38.859526 (+ 3755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:38.861371 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:38.862001 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:38.862112 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:38.865137 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:51:38.865142 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:51:38.865142 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:51:38.865160 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:51:38.867100 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:38.867602 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:38.867605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:38.867606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:38.867710 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:38.868015 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:38.868016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:38.868016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:38.868263 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:51:38.868372 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:38.868690 (+ 318us) server_negotiation.cc:299] Negotiation successful
0928 22:51:38.868840 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1407,"thread_start_us":86,"threads_started":1}
I20260928 22:51:38.869053 31357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:38.852776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.129:56527)
0928 22:51:38.853155 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:38.853215 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 22:51:38.854053 (+ 838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:51:38.854927 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:51:38.854932 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:51:38.855328 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:51:38.855607 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:38.855614 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:38.859685 (+ 4071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:38.859687 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:38.861263 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:38.861266 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:38.862124 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:38.862126 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:38.862305 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:38.862980 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:51:38.862991 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:51:38.865003 (+ 2012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:51:38.867229 (+ 2226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:38.867238 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:38.867240 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:38.867502 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:38.867808 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:38.867809 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:38.867809 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:38.867937 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:38.868347 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:51:38.868354 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:51:38.868509 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 22:51:38.868633 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 22:51:38.868836 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":108,"threads_started":1}
I20260928 22:51:38.870224 31355 heartbeater.cc:347] Connected to a master server at 127.16.121.190:37803
I20260928 22:51:38.870362 31355 heartbeater.cc:464] Registering TS with master...
I20260928 22:51:38.870559 31355 heartbeater.cc:511] Master 127.16.121.190:37803 requested a full tablet report, sending...
I20260928 22:51:38.871404 31167 ts_manager.cc:194] Registered new tserver with Master: 86d13097cb5348d2a58f4c202689f82e (127.16.121.129:36367)
I20260928 22:51:38.872607 31167 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.129@KRBTEST.COM'} at 127.16.121.129:56527
I20260928 22:51:38.873705 31355 heartbeater.cc:503] Master 127.16.121.190:37803 was elected leader, sending a full tablet report...
Sep 28 22:51:38 dist-test-slave-56vw krb5kdc[30586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635898, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for kudu/127.16.121.129@KRBTEST.COM
I20260928 22:51:38.881367 31363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:38.872107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58960 (local address 127.16.121.129:36367)
0928 22:51:38.872286 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:38.872290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:38.872450 (+ 160us) server_negotiation.cc:407] Connection header received
0928 22:51:38.872580 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:38.872582 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:38.872597 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:38.872691 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:51:38.873401 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:38.874496 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:38.875463 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:38.875810 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:38.875901 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:38.878319 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:51:38.878329 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:51:38.878329 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:51:38.878337 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:51:38.880017 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:38.880444 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:38.880448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:38.880448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:38.880523 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:38.880728 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:38.880729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:38.880730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:38.880890 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:51:38.880985 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:38.881167 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 22:51:38.881265 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":81,"threads_started":1}
I20260928 22:51:38.881377 31362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:38.872049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:36367 (local address 127.0.0.1:58960)
0928 22:51:38.872345 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:38.872375 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 22:51:38.872472 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:51:38.872827 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:51:38.872833 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:51:38.873231 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:51:38.873296 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:38.873299 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:38.874591 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:38.874592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:38.875356 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:38.875358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:38.875900 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:38.875901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:38.875995 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:38.876563 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:51:38.876568 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:51:38.878218 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:51:38.880124 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:38.880132 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:38.880134 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:38.880356 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:38.880594 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:38.880595 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:38.880596 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:38.880663 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:38.880962 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:51:38.880970 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:51:38.881025 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 22:51:38.881131 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 22:51:38.881265 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":106,"threads_started":1}
I20260928 22:51:38.882090 31289 tablet_service.cc:1514] Processing DeleteTablet for tablet df8170eee726457ea285c53bc5b909ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:51:38 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:58960
I20260928 22:51:38.882755 31364 tablet_replica.cc:330] stopping tablet replica
I20260928 22:51:38.882953 31364 raft_consensus.cc:2229] T df8170eee726457ea285c53bc5b909ae P 86d13097cb5348d2a58f4c202689f82e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:51:38.883044 31364 raft_consensus.cc:2258] T df8170eee726457ea285c53bc5b909ae P 86d13097cb5348d2a58f4c202689f82e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:51:38.883397 31364 ts_tablet_manager.cc:1905] T df8170eee726457ea285c53bc5b909ae P 86d13097cb5348d2a58f4c202689f82e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:51:38.885715 31364 ts_tablet_manager.cc:1918] T df8170eee726457ea285c53bc5b909ae P 86d13097cb5348d2a58f4c202689f82e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:51:38.885833 31364 log.cc:1196] T df8170eee726457ea285c53bc5b909ae P 86d13097cb5348d2a58f4c202689f82e: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-0/wal/wals/df8170eee726457ea285c53bc5b909ae
I20260928 22:51:38.886058 31364 ts_tablet_manager.cc:1939] T df8170eee726457ea285c53bc5b909ae P 86d13097cb5348d2a58f4c202689f82e: Deleting consensus metadata
I20260928 22:51:38.886618 31154 catalog_manager.cc:4841] TS 86d13097cb5348d2a58f4c202689f82e (127.16.121.129:36367): tablet df8170eee726457ea285c53bc5b909ae (table test-table [id=0bd344fa86a2463990916adcc2eabf0b]) successfully deleted
W20260928 22:51:38.947242 31360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:51:38.947440 31360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:51:38.947471 31360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:51:38.948568 31360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:51:38.948603 31360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:51:38.948628 31360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.130
I20260928 22:51:38.950139 31360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:36021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab
--principal=kudu/127.16.121.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.130:41643
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.121.130
--webserver_port=42105
--tserver_master_addrs=127.16.121.190:37803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.130
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:51:38.950330 31360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:51:38.950511 31360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 22:51:38.955847 31360 server_base.cc:1034] running on GCE node
W20260928 22:51:38.955791 31372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:51:38.955869 31369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:51:38.955894 31370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:51:38.956194 31360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:51:38.956450 31360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:51:38.957599 31360 hybrid_clock.cc:648] HybridClock initialized: now 1790635898957588 us; error 31 us; skew 500 ppm
Sep 28 22:51:38 dist-test-slave-56vw krb5kdc[30586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635898, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:51:38.960206 31360 init.cc:377] Logged in from keytab as kudu/127.16.121.130@KRBTEST.COM (short username kudu)
I20260928 22:51:38.960682 31360 webserver.cc:460] Webserver started at http://127.16.121.130:42105/ using document root <none> and password file <none>
I20260928 22:51:38.960882 31360 fs_manager.cc:359] Metadata directory not provided
I20260928 22:51:38.960932 31360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:51:38.962237 31360 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:51:38.962888 31379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:38.963126 31360 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:51:38.963228 31360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-1/wal
uuid: "90486aa702d140f19e71be00c5088990"
format_stamp: "Formatted at 2026-09-28 22:51:38 on dist-test-slave-56vw"
server_key: "c5b4f7ce9c47173fe00ad0a32ca73011"
server_key_iv: "627c6ae39d4c7a429c96adddce7d38cd"
server_key_version: "encryptionkey@0"
I20260928 22:51:38.963603 31360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:51:38.977074 31360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:51:38.979597 31360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:51:38.979776 31360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:51:38.980054 31360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:51:38.980509 31386 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260928 22:51:38.982170 31386 ts_tablet_manager.cc:1852] T df8170eee726457ea285c53bc5b909ae P 90486aa702d140f19e71be00c5088990: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260928 22:51:38.982477 31386 ts_tablet_manager.cc:1905] T df8170eee726457ea285c53bc5b909ae P 90486aa702d140f19e71be00c5088990: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:51:38.984951 31386 ts_tablet_manager.cc:1918] T df8170eee726457ea285c53bc5b909ae P 90486aa702d140f19e71be00c5088990: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:51:38.985038 31386 log.cc:1196] T df8170eee726457ea285c53bc5b909ae P 90486aa702d140f19e71be00c5088990: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-1/wal/wals/df8170eee726457ea285c53bc5b909ae
I20260928 22:51:38.985252 31386 ts_tablet_manager.cc:1939] T df8170eee726457ea285c53bc5b909ae P 90486aa702d140f19e71be00c5088990: Deleting consensus metadata
I20260928 22:51:38.985423 31360 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260928 22:51:38.985503 31360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260928 22:51:38.985544 31360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:51:38.985567 31360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:38.991748 31360 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.130:41643
I20260928 22:51:38.991809 31493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.130:41643 every 8 connection(s)
I20260928 22:51:38.992146 31360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
I20260928 22:51:38.995769 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 31360
I20260928 22:51:38.996042 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.131:39589
--local_ip_for_outbound_sockets=127.16.121.131
--tserver_master_addrs=127.16.121.190:37803
--webserver_port=45171
--webserver_interface=127.16.121.131
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab
--principal=kudu/127.16.121.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:36021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 22:51:38 dist-test-slave-56vw krb5kdc[30586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635898, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.130@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:51:39.002502 31358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:38.992907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:37801 (local address 127.16.121.190:37803)
0928 22:51:38.992976 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:38.992979 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:38.993284 (+ 305us) server_negotiation.cc:407] Connection header received
0928 22:51:38.993823 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:38.993824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:38.993838 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:38.993897 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:51:38.994570 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:38.995105 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:38.996283 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:38.996638 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:38.996715 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:38.999093 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:51:38.999095 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:51:38.999096 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:51:38.999104 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:51:39.000925 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:39.001385 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:39.001387 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:39.001387 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:39.001455 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:39.001737 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:39.001738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:39.001739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:39.001906 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:51:39.001986 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:39.002253 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 22:51:39.002363 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 22:51:39.002513 31496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:38.992916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.130:37801)
0928 22:51:38.993139 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:38.993182 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 22:51:38.993727 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:51:38.993986 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:51:38.993990 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:51:38.994256 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:51:38.994481 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:38.994485 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:38.995193 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:38.995194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:38.996123 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:38.996127 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:38.996726 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:38.996727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:38.996843 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:38.997461 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:51:38.997470 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:51:38.998972 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:51:39.001024 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:39.001030 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:39.001032 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:39.001300 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:39.001547 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:39.001548 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:39.001548 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:39.001658 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:39.001983 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:51:39.001988 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:51:39.002112 (+ 124us) client_negotiation.cc:769] Sending connection context
0928 22:51:39.002209 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 22:51:39.002345 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":86,"threads_started":1}
I20260928 22:51:39.003109 31494 heartbeater.cc:347] Connected to a master server at 127.16.121.190:37803
I20260928 22:51:39.003226 31494 heartbeater.cc:464] Registering TS with master...
I20260928 22:51:39.003429 31494 heartbeater.cc:511] Master 127.16.121.190:37803 requested a full tablet report, sending...
I20260928 22:51:39.003898 31167 ts_manager.cc:194] Registered new tserver with Master: 90486aa702d140f19e71be00c5088990 (127.16.121.130:41643)
I20260928 22:51:39.004308 31167 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.130@KRBTEST.COM'} at 127.16.121.130:37801
W20260928 22:51:39.079978 31497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:51:39.080233 31497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:51:39.080283 31497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:51:39.081475 31497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:51:39.081548 31497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:51:39.081593 31497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.131
I20260928 22:51:39.083091 31497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:36021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab
--principal=kudu/127.16.121.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.131:39589
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.121.131
--webserver_port=45171
--tserver_master_addrs=127.16.121.190:37803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31497
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.131
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:51:39.083382 31497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:51:39.083621 31497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 22:51:39.088773 31497 server_base.cc:1034] running on GCE node
W20260928 22:51:39.088711 31502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:51:39.088722 31505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:51:39.088742 31503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:51:39.089164 31497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:51:39.089407 31497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:51:39.090602 31497 hybrid_clock.cc:648] HybridClock initialized: now 1790635899090568 us; error 43 us; skew 500 ppm
Sep 28 22:51:39 dist-test-slave-56vw krb5kdc[30586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635899, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:51:39.093724 31497 init.cc:377] Logged in from keytab as kudu/127.16.121.131@KRBTEST.COM (short username kudu)
I20260928 22:51:39.094221 31497 webserver.cc:460] Webserver started at http://127.16.121.131:45171/ using document root <none> and password file <none>
I20260928 22:51:39.094478 31497 fs_manager.cc:359] Metadata directory not provided
I20260928 22:51:39.094560 31497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:51:39.096187 31497 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:51:39.096953 31512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:39.097175 31497 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:51:39.097277 31497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-2/wal
uuid: "87072ae366cd40b48beee55f28afb7b9"
format_stamp: "Formatted at 2026-09-28 22:51:38 on dist-test-slave-56vw"
server_key: "731b56b30449e31d337fb18fa7a0f486"
server_key_iv: "840d244dc374d7f4e039b7ed8a04cc69"
server_key_version: "encryptionkey@0"
I20260928 22:51:39.097663 31497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:51:39.109122 31497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:51:39.111718 31497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:51:39.111902 31497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:51:39.112164 31497 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:51:39.112498 31497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:51:39.112541 31497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:39.112596 31497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:51:39.112625 31497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:39.119365 31497 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.131:39589
I20260928 22:51:39.119483 31625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.131:39589 every 8 connection(s)
I20260928 22:51:39.119846 31497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
I20260928 22:51:39.121997 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 31497
Sep 28 22:51:39 dist-test-slave-56vw krb5kdc[30586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635899, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.131@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:51:39.129357 31630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:39.122409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59894 (local address 127.16.121.190:37803)
0928 22:51:39.123071 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:39.123073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:39.123079 (+ 6us) server_negotiation.cc:407] Connection header received
0928 22:51:39.123097 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:39.123098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:39.123123 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:39.123183 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:51:39.123767 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:39.124260 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:39.124997 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:39.125190 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:39.125268 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:39.126472 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:51:39.126473 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:51:39.126474 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:51:39.126481 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:51:39.128123 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:39.128507 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:39.128509 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:39.128509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:39.128560 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:39.128770 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:39.128771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:39.128771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:39.128903 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:51:39.129109 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:39.129146 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 22:51:39.129226 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":48,"threads_started":1}
I20260928 22:51:39.130403 31358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:39.120662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:55717 (local address 127.16.121.190:37803)
0928 22:51:39.120775 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:39.120777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:39.120960 (+ 183us) server_negotiation.cc:407] Connection header received
0928 22:51:39.121532 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:39.121534 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:39.121548 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:39.121599 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:51:39.123004 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:39.123623 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:39.124654 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:39.124877 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:39.124965 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:39.127423 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:51:39.127425 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:51:39.127425 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:51:39.127433 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:51:39.128903 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:39.129400 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:39.129401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:39.129401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:39.129449 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:39.129759 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:39.129760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:39.129760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:39.129872 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:51:39.129948 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:39.130234 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 22:51:39.130313 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 22:51:39.130623 31628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:39.120629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.131:55717)
0928 22:51:39.120833 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:39.120871 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 22:51:39.121411 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:51:39.121683 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:51:39.121687 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:51:39.122667 (+ 980us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:51:39.122917 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:39.122923 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:39.123718 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:39.123719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:39.124560 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:39.124562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:39.124974 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:39.124976 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:39.125111 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:39.125674 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:51:39.125693 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:51:39.127320 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:51:39.129010 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:39.129017 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:39.129019 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:39.129246 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:39.129540 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:39.129541 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:39.129541 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:39.129680 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:39.129941 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:51:39.129948 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:51:39.130095 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 22:51:39.130202 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 22:51:39.130380 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":95,"threads_started":1}
I20260928 22:51:39.131114 31626 heartbeater.cc:347] Connected to a master server at 127.16.121.190:37803
I20260928 22:51:39.131237 31626 heartbeater.cc:464] Registering TS with master...
I20260928 22:51:39.131423 31626 heartbeater.cc:511] Master 127.16.121.190:37803 requested a full tablet report, sending...
I20260928 22:51:39.131853 31167 ts_manager.cc:194] Registered new tserver with Master: 87072ae366cd40b48beee55f28afb7b9 (127.16.121.131:39589)
I20260928 22:51:39.132346 31167 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.131@KRBTEST.COM'} at 127.16.121.131:55717
I20260928 22:51:39.141193 16870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:51:39.144047 31358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:39.141603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59910 (local address 127.16.121.190:37803)
0928 22:51:39.141650 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:39.141654 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:39.141720 (+ 66us) server_negotiation.cc:407] Connection header received
0928 22:51:39.141815 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:39.141816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:39.141853 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:39.141914 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:51:39.142042 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:39.142562 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:39.143397 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:39.143584 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:39.143656 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:39.143778 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:51:39.143884 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:51:39.143949 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:39.143970 (+ 21us) server_negotiation.cc:299] Negotiation successful
0928 22:51:39.143988 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:51:39.144381 31167 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:59910:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:51:39.146076 31167 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:51:39.148473 31289 tablet_service.cc:1467] Processing CreateTablet for tablet 87b9ff3902364b85a0de9c8dc5d305e5 (DEFAULT_TABLE table=test-table [id=20a3d436d145449d84e79c7195e4db61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:51:39.148689 31289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87b9ff3902364b85a0de9c8dc5d305e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:51:39.150080 31247 tablet_bootstrap.cc:492] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e: Bootstrap starting.
I20260928 22:51:39.150417 31247 tablet_bootstrap.cc:654] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e: Neither blocks nor log segments found. Creating new log.
I20260928 22:51:39.151515 31247 tablet_bootstrap.cc:492] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e: No bootstrap required, opened a new log
I20260928 22:51:39.151571 31247 ts_tablet_manager.cc:1397] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:51:39.151705 31247 raft_consensus.cc:348] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90486aa702d140f19e71be00c5088990" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 41643 } } peers { permanent_uuid: "87072ae366cd40b48beee55f28afb7b9" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 39589 } } peers { permanent_uuid: "86d13097cb5348d2a58f4c202689f82e" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 36367 } }
I20260928 22:51:39.151777 31247 raft_consensus.cc:374] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:51:39.151830 31247 raft_consensus.cc:729] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86d13097cb5348d2a58f4c202689f82e, State: Initialized, Role: FOLLOWER
I20260928 22:51:39.151882 31247 consensus_queue.cc:260] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90486aa702d140f19e71be00c5088990" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 41643 } } peers { permanent_uuid: "87072ae366cd40b48beee55f28afb7b9" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 39589 } } peers { permanent_uuid: "86d13097cb5348d2a58f4c202689f82e" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 36367 } }
I20260928 22:51:39.151985 31247 ts_tablet_manager.cc:1428] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:39.152045 31362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:39.148166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.131:39589 (local address 127.0.0.1:55970)
0928 22:51:39.148203 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:39.148216 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:51:39.148255 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:51:39.148839 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:51:39.148840 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:51:39.148842 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:51:39.148892 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:39.148894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:39.149857 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:39.149859 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:39.150816 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:39.150818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:39.151745 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:39.151746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:39.151847 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:39.151869 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 22:51:39.151923 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 22:51:39.151974 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260928 22:51:39.152772 31632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:39.148309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55970 (local address 127.16.121.131:39589)
0928 22:51:39.148642 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:39.148647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:39.148657 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:51:39.148699 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:39.148700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:39.148713 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:39.148835 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:51:39.148963 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:39.149750 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:39.150903 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:39.151677 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:39.151748 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:39.151807 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:39.152020 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 22:51:39.152299 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":85,"threads_started":1}
I20260928 22:51:39.153543 31560 tablet_service.cc:1467] Processing CreateTablet for tablet 87b9ff3902364b85a0de9c8dc5d305e5 (DEFAULT_TABLE table=test-table [id=20a3d436d145449d84e79c7195e4db61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:51:39.153920 31560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87b9ff3902364b85a0de9c8dc5d305e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:51:39.154084 31633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:39.148166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.130:41643 (local address 127.0.0.1:57938)
0928 22:51:39.148459 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:39.148468 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:51:39.148519 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:51:39.150537 (+ 2018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:51:39.150538 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:51:39.150540 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:51:39.150576 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:39.150577 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:39.151873 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:39.151874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:39.153008 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:39.153011 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:39.153847 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:39.153848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:39.153906 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:39.153911 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:51:39.153969 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 22:51:39.154010 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":99,"thread_start_us":124,"threads_started":1}
I20260928 22:51:39.154096 31634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:39.148295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57938 (local address 127.16.121.130:41643)
0928 22:51:39.150310 (+ 2015us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:39.150315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:39.150326 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:51:39.150364 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:39.150366 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:39.150384 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:39.150492 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:51:39.150654 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:39.151754 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:39.153089 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:39.153761 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:39.153847 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:39.153905 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:39.153982 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 22:51:39.154021 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1975,"thread_start_us":108,"threads_started":1}
I20260928 22:51:39.155610 31428 tablet_service.cc:1467] Processing CreateTablet for tablet 87b9ff3902364b85a0de9c8dc5d305e5 (DEFAULT_TABLE table=test-table [id=20a3d436d145449d84e79c7195e4db61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:51:39.155937 31428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87b9ff3902364b85a0de9c8dc5d305e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:51:39.157507 31636 tablet_bootstrap.cc:492] T 87b9ff3902364b85a0de9c8dc5d305e5 P 87072ae366cd40b48beee55f28afb7b9: Bootstrap starting.
I20260928 22:51:39.158087 31386 tablet_bootstrap.cc:492] T 87b9ff3902364b85a0de9c8dc5d305e5 P 90486aa702d140f19e71be00c5088990: Bootstrap starting.
I20260928 22:51:39.158239 31636 tablet_bootstrap.cc:654] T 87b9ff3902364b85a0de9c8dc5d305e5 P 87072ae366cd40b48beee55f28afb7b9: Neither blocks nor log segments found. Creating new log.
I20260928 22:51:39.158830 31636 log.cc:824] T 87b9ff3902364b85a0de9c8dc5d305e5 P 87072ae366cd40b48beee55f28afb7b9: Log is configured to *not* fsync() on all Append() calls
I20260928 22:51:39.158857 31386 tablet_bootstrap.cc:654] T 87b9ff3902364b85a0de9c8dc5d305e5 P 90486aa702d140f19e71be00c5088990: Neither blocks nor log segments found. Creating new log.
I20260928 22:51:39.159212 31386 log.cc:824] T 87b9ff3902364b85a0de9c8dc5d305e5 P 90486aa702d140f19e71be00c5088990: Log is configured to *not* fsync() on all Append() calls
I20260928 22:51:39.159694 31636 tablet_bootstrap.cc:492] T 87b9ff3902364b85a0de9c8dc5d305e5 P 87072ae366cd40b48beee55f28afb7b9: No bootstrap required, opened a new log
I20260928 22:51:39.159776 31636 ts_tablet_manager.cc:1397] T 87b9ff3902364b85a0de9c8dc5d305e5 P 87072ae366cd40b48beee55f28afb7b9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 22:51:39.159978 31386 tablet_bootstrap.cc:492] T 87b9ff3902364b85a0de9c8dc5d305e5 P 90486aa702d140f19e71be00c5088990: No bootstrap required, opened a new log
I20260928 22:51:39.160036 31386 ts_tablet_manager.cc:1397] T 87b9ff3902364b85a0de9c8dc5d305e5 P 90486aa702d140f19e71be00c5088990: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 22:51:39.161929 31636 raft_consensus.cc:348] T 87b9ff3902364b85a0de9c8dc5d305e5 P 87072ae366cd40b48beee55f28afb7b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90486aa702d140f19e71be00c5088990" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 41643 } } peers { permanent_uuid: "87072ae366cd40b48beee55f28afb7b9" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 39589 } } peers { permanent_uuid: "86d13097cb5348d2a58f4c202689f82e" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 36367 } }
I20260928 22:51:39.161929 31386 raft_consensus.cc:348] T 87b9ff3902364b85a0de9c8dc5d305e5 P 90486aa702d140f19e71be00c5088990 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90486aa702d140f19e71be00c5088990" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 41643 } } peers { permanent_uuid: "87072ae366cd40b48beee55f28afb7b9" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 39589 } } peers { permanent_uuid: "86d13097cb5348d2a58f4c202689f82e" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 36367 } }
I20260928 22:51:39.162110 31386 raft_consensus.cc:374] T 87b9ff3902364b85a0de9c8dc5d305e5 P 90486aa702d140f19e71be00c5088990 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:51:39.162111 31636 raft_consensus.cc:374] T 87b9ff3902364b85a0de9c8dc5d305e5 P 87072ae366cd40b48beee55f28afb7b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:51:39.162147 31636 raft_consensus.cc:729] T 87b9ff3902364b85a0de9c8dc5d305e5 P 87072ae366cd40b48beee55f28afb7b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87072ae366cd40b48beee55f28afb7b9, State: Initialized, Role: FOLLOWER
I20260928 22:51:39.162147 31386 raft_consensus.cc:729] T 87b9ff3902364b85a0de9c8dc5d305e5 P 90486aa702d140f19e71be00c5088990 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90486aa702d140f19e71be00c5088990, State: Initialized, Role: FOLLOWER
I20260928 22:51:39.162273 31386 consensus_queue.cc:260] T 87b9ff3902364b85a0de9c8dc5d305e5 P 90486aa702d140f19e71be00c5088990 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90486aa702d140f19e71be00c5088990" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 41643 } } peers { permanent_uuid: "87072ae366cd40b48beee55f28afb7b9" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 39589 } } peers { permanent_uuid: "86d13097cb5348d2a58f4c202689f82e" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 36367 } }
I20260928 22:51:39.162273 31636 consensus_queue.cc:260] T 87b9ff3902364b85a0de9c8dc5d305e5 P 87072ae366cd40b48beee55f28afb7b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90486aa702d140f19e71be00c5088990" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 41643 } } peers { permanent_uuid: "87072ae366cd40b48beee55f28afb7b9" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 39589 } } peers { permanent_uuid: "86d13097cb5348d2a58f4c202689f82e" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 36367 } }
I20260928 22:51:39.162568 31636 ts_tablet_manager.cc:1428] T 87b9ff3902364b85a0de9c8dc5d305e5 P 87072ae366cd40b48beee55f28afb7b9: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 22:51:39.162568 31386 ts_tablet_manager.cc:1428] T 87b9ff3902364b85a0de9c8dc5d305e5 P 90486aa702d140f19e71be00c5088990: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 22:51:39.162616 31626 heartbeater.cc:503] Master 127.16.121.190:37803 was elected leader, sending a full tablet report...
I20260928 22:51:39.162858 31494 heartbeater.cc:503] Master 127.16.121.190:37803 was elected leader, sending a full tablet report...
W20260928 22:51:39.242733 31495 tablet.cc:2369] T 87b9ff3902364b85a0de9c8dc5d305e5 P 90486aa702d140f19e71be00c5088990: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:51:39.256909 31639 raft_consensus.cc:482] T 87b9ff3902364b85a0de9c8dc5d305e5 P 90486aa702d140f19e71be00c5088990 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:51:39.257072 31639 raft_consensus.cc:504] T 87b9ff3902364b85a0de9c8dc5d305e5 P 90486aa702d140f19e71be00c5088990 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90486aa702d140f19e71be00c5088990" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 41643 } } peers { permanent_uuid: "87072ae366cd40b48beee55f28afb7b9" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 39589 } } peers { permanent_uuid: "86d13097cb5348d2a58f4c202689f82e" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 36367 } }
I20260928 22:51:39.257464 31639 leader_election.cc:288] T 87b9ff3902364b85a0de9c8dc5d305e5 P 90486aa702d140f19e71be00c5088990 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87072ae366cd40b48beee55f28afb7b9 (127.16.121.131:39589), 86d13097cb5348d2a58f4c202689f82e (127.16.121.129:36367)
I20260928 22:51:39.261492 31632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:39.257733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:48843 (local address 127.16.121.131:39589)
0928 22:51:39.257803 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:39.257805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:39.257960 (+ 155us) server_negotiation.cc:407] Connection header received
0928 22:51:39.258014 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:39.258015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:39.258024 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:39.258073 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:51:39.258253 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:39.258965 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:39.260525 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:39.261100 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:39.261185 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:39.261195 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:39.261370 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 22:51:39.261423 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 22:51:39.261502 31496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:39.257683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:36367 (local address 127.16.121.130:33415)
0928 22:51:39.257738 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:39.257751 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:51:39.257797 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:51:39.258000 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:51:39.258001 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:51:39.258003 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:51:39.258100 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:39.258102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:39.259080 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:39.259081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:39.260410 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:39.260413 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:39.261274 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:39.261274 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:39.261348 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:39.261352 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:51:39.261406 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 22:51:39.261436 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":8}
I20260928 22:51:39.261530 31363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:39.257754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:33415 (local address 127.16.121.129:36367)
0928 22:51:39.257818 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:39.257820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:39.257828 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:51:39.257858 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:39.257859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:39.257890 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:39.257972 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:51:39.258179 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:39.258973 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:39.260526 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:39.261192 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:39.261263 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:39.261351 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:39.261417 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 22:51:39.261460 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 22:51:39.261512 31641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:39.257690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.131:39589 (local address 127.16.121.130:48843)
0928 22:51:39.257869 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:39.257879 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:51:39.257941 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:51:39.258099 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:51:39.258099 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:51:39.258100 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:51:39.258174 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:39.258176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:39.259080 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:39.259081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:39.260410 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:39.260413 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:39.261193 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:39.261194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:39.261282 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:39.261287 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:51:39.261346 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 22:51:39.261398 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":119,"thread_start_us":104,"threads_started":1}
I20260928 22:51:39.261857 31309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87b9ff3902364b85a0de9c8dc5d305e5" candidate_uuid: "90486aa702d140f19e71be00c5088990" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86d13097cb5348d2a58f4c202689f82e" is_pre_election: true
I20260928 22:51:39.262079 31309 raft_consensus.cc:2454] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90486aa702d140f19e71be00c5088990 in term 0.
I20260928 22:51:39.262106 31580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87b9ff3902364b85a0de9c8dc5d305e5" candidate_uuid: "90486aa702d140f19e71be00c5088990" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87072ae366cd40b48beee55f28afb7b9" is_pre_election: true
I20260928 22:51:39.262248 31580 raft_consensus.cc:2454] T 87b9ff3902364b85a0de9c8dc5d305e5 P 87072ae366cd40b48beee55f28afb7b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90486aa702d140f19e71be00c5088990 in term 0.
I20260928 22:51:39.262346 31383 leader_election.cc:302] T 87b9ff3902364b85a0de9c8dc5d305e5 P 90486aa702d140f19e71be00c5088990 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86d13097cb5348d2a58f4c202689f82e, 90486aa702d140f19e71be00c5088990; no voters:
I20260928 22:51:39.262516 31639 raft_consensus.cc:2781] T 87b9ff3902364b85a0de9c8dc5d305e5 P 90486aa702d140f19e71be00c5088990 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:51:39.262559 31639 raft_consensus.cc:482] T 87b9ff3902364b85a0de9c8dc5d305e5 P 90486aa702d140f19e71be00c5088990 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:51:39.262575 31639 raft_consensus.cc:3037] T 87b9ff3902364b85a0de9c8dc5d305e5 P 90486aa702d140f19e71be00c5088990 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:51:39.263341 31639 raft_consensus.cc:504] T 87b9ff3902364b85a0de9c8dc5d305e5 P 90486aa702d140f19e71be00c5088990 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90486aa702d140f19e71be00c5088990" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 41643 } } peers { permanent_uuid: "87072ae366cd40b48beee55f28afb7b9" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 39589 } } peers { permanent_uuid: "86d13097cb5348d2a58f4c202689f82e" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 36367 } }
I20260928 22:51:39.263576 31639 leader_election.cc:288] T 87b9ff3902364b85a0de9c8dc5d305e5 P 90486aa702d140f19e71be00c5088990 [CANDIDATE]: Term 1 election: Requested vote from peers 87072ae366cd40b48beee55f28afb7b9 (127.16.121.131:39589), 86d13097cb5348d2a58f4c202689f82e (127.16.121.129:36367)
I20260928 22:51:39.263684 31580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87b9ff3902364b85a0de9c8dc5d305e5" candidate_uuid: "90486aa702d140f19e71be00c5088990" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87072ae366cd40b48beee55f28afb7b9"
I20260928 22:51:39.263727 31309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87b9ff3902364b85a0de9c8dc5d305e5" candidate_uuid: "90486aa702d140f19e71be00c5088990" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86d13097cb5348d2a58f4c202689f82e"
I20260928 22:51:39.263768 31580 raft_consensus.cc:3037] T 87b9ff3902364b85a0de9c8dc5d305e5 P 87072ae366cd40b48beee55f28afb7b9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:51:39.263808 31309 raft_consensus.cc:3037] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:51:39.264668 31580 raft_consensus.cc:2454] T 87b9ff3902364b85a0de9c8dc5d305e5 P 87072ae366cd40b48beee55f28afb7b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90486aa702d140f19e71be00c5088990 in term 1.
I20260928 22:51:39.264668 31309 raft_consensus.cc:2454] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90486aa702d140f19e71be00c5088990 in term 1.
I20260928 22:51:39.264889 31382 leader_election.cc:302] T 87b9ff3902364b85a0de9c8dc5d305e5 P 90486aa702d140f19e71be00c5088990 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87072ae366cd40b48beee55f28afb7b9, 90486aa702d140f19e71be00c5088990; no voters:
I20260928 22:51:39.265015 31639 raft_consensus.cc:2781] T 87b9ff3902364b85a0de9c8dc5d305e5 P 90486aa702d140f19e71be00c5088990 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:51:39.265197 31639 raft_consensus.cc:686] T 87b9ff3902364b85a0de9c8dc5d305e5 P 90486aa702d140f19e71be00c5088990 [term 1 LEADER]: Becoming Leader. State: Replica: 90486aa702d140f19e71be00c5088990, State: Running, Role: LEADER
I20260928 22:51:39.265337 31639 consensus_queue.cc:237] T 87b9ff3902364b85a0de9c8dc5d305e5 P 90486aa702d140f19e71be00c5088990 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90486aa702d140f19e71be00c5088990" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 41643 } } peers { permanent_uuid: "87072ae366cd40b48beee55f28afb7b9" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 39589 } } peers { permanent_uuid: "86d13097cb5348d2a58f4c202689f82e" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 36367 } }
I20260928 22:51:39.266237 31167 catalog_manager.cc:5495] T 87b9ff3902364b85a0de9c8dc5d305e5 P 90486aa702d140f19e71be00c5088990 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90486aa702d140f19e71be00c5088990 (127.16.121.130). New cstate: current_term: 1 leader_uuid: "90486aa702d140f19e71be00c5088990" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90486aa702d140f19e71be00c5088990" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 41643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87072ae366cd40b48beee55f28afb7b9" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 39589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86d13097cb5348d2a58f4c202689f82e" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 36367 } health_report { overall_health: UNKNOWN } } }
I20260928 22:51:39.300549 31634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:39.297938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57948 (local address 127.16.121.130:41643)
0928 22:51:39.298023 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:39.298025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:39.298036 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:51:39.298073 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:39.298074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:39.298095 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:39.298192 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:51:39.298299 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:39.298876 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:39.299668 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:39.299954 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:39.300019 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:39.300221 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:51:39.300336 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:51:39.300427 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:39.300444 (+ 17us) server_negotiation.cc:299] Negotiation successful
0928 22:51:39.300464 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 22:51:39.302680 31309 raft_consensus.cc:1261] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [term 1 FOLLOWER]: Refusing update from remote peer 90486aa702d140f19e71be00c5088990: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:51:39.302690 31580 raft_consensus.cc:1261] T 87b9ff3902364b85a0de9c8dc5d305e5 P 87072ae366cd40b48beee55f28afb7b9 [term 1 FOLLOWER]: Refusing update from remote peer 90486aa702d140f19e71be00c5088990: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:51:39.303164 31642 consensus_queue.cc:1035] T 87b9ff3902364b85a0de9c8dc5d305e5 P 90486aa702d140f19e71be00c5088990 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87072ae366cd40b48beee55f28afb7b9" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 39589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:51:39.303313 31639 consensus_queue.cc:1035] T 87b9ff3902364b85a0de9c8dc5d305e5 P 90486aa702d140f19e71be00c5088990 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86d13097cb5348d2a58f4c202689f82e" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 36367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:51:39.305727 31646 mvcc.cc:205] Tried to move back new op lower bound from 7334444643540426752 to 7334444643391528960. Current Snapshot: MvccSnapshot[applied={T|T < 7334444643540426752}]
I20260928 22:51:39.305636 31645 mvcc.cc:205] Tried to move back new op lower bound from 7334444643540426752 to 7334444643391528960. Current Snapshot: MvccSnapshot[applied={T|T < 7334444643540426752}]
I20260928 22:51:39.308943 31167 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:59910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:51:39.309052 31167 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:59910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:51:39.309929 31167 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447: Sending DeleteTablet for 3 replicas of tablet 87b9ff3902364b85a0de9c8dc5d305e5
I20260928 22:51:39.310254 31428 tablet_service.cc:1514] Processing DeleteTablet for tablet 87b9ff3902364b85a0de9c8dc5d305e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:51:39 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:57938
I20260928 22:51:39.310303 31560 tablet_service.cc:1514] Processing DeleteTablet for tablet 87b9ff3902364b85a0de9c8dc5d305e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:51:39 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:55970
I20260928 22:51:39.310472 31289 tablet_service.cc:1514] Processing DeleteTablet for tablet 87b9ff3902364b85a0de9c8dc5d305e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:51:39 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:58960
I20260928 22:51:39.310509 31655 tablet_replica.cc:330] T 87b9ff3902364b85a0de9c8dc5d305e5 P 90486aa702d140f19e71be00c5088990: stopping tablet replica
I20260928 22:51:39.310572 31364 tablet_replica.cc:330] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e: stopping tablet replica
I20260928 22:51:39.310606 31655 raft_consensus.cc:2229] T 87b9ff3902364b85a0de9c8dc5d305e5 P 90486aa702d140f19e71be00c5088990 [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:51:39.310627 31364 raft_consensus.cc:2229] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:51:39.310678 31364 raft_consensus.cc:2258] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:51:39.310717 31655 raft_consensus.cc:2258] T 87b9ff3902364b85a0de9c8dc5d305e5 P 90486aa702d140f19e71be00c5088990 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:51:39.311029 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 31205
I20260928 22:51:39.311031 31656 tablet_replica.cc:330] T 87b9ff3902364b85a0de9c8dc5d305e5 P 87072ae366cd40b48beee55f28afb7b9: stopping tablet replica
I20260928 22:51:39.311033 31364 ts_tablet_manager.cc:1905] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:51:39.311650 31656 raft_consensus.cc:2229] T 87b9ff3902364b85a0de9c8dc5d305e5 P 87072ae366cd40b48beee55f28afb7b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:51:39.311785 31656 raft_consensus.cc:2258] T 87b9ff3902364b85a0de9c8dc5d305e5 P 87072ae366cd40b48beee55f28afb7b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:51:39.311802 31655 ts_tablet_manager.cc:1905] T 87b9ff3902364b85a0de9c8dc5d305e5 P 90486aa702d140f19e71be00c5088990: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:51:39.312223 31656 ts_tablet_manager.cc:1905] T 87b9ff3902364b85a0de9c8dc5d305e5 P 87072ae366cd40b48beee55f28afb7b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:51:39.313372 31655 ts_tablet_manager.cc:1918] T 87b9ff3902364b85a0de9c8dc5d305e5 P 90486aa702d140f19e71be00c5088990: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:51:39.313426 31655 log.cc:1196] T 87b9ff3902364b85a0de9c8dc5d305e5 P 90486aa702d140f19e71be00c5088990: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-1/wal/wals/87b9ff3902364b85a0de9c8dc5d305e5
I20260928 22:51:39.313655 31655 ts_tablet_manager.cc:1939] T 87b9ff3902364b85a0de9c8dc5d305e5 P 90486aa702d140f19e71be00c5088990: Deleting consensus metadata
I20260928 22:51:39.314049 31152 catalog_manager.cc:4841] TS 90486aa702d140f19e71be00c5088990 (127.16.121.130:41643): tablet 87b9ff3902364b85a0de9c8dc5d305e5 (table test-table [id=20a3d436d145449d84e79c7195e4db61]) successfully deleted
I20260928 22:51:39.314590 31656 ts_tablet_manager.cc:1918] T 87b9ff3902364b85a0de9c8dc5d305e5 P 87072ae366cd40b48beee55f28afb7b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:51:39.314666 31656 log.cc:1196] T 87b9ff3902364b85a0de9c8dc5d305e5 P 87072ae366cd40b48beee55f28afb7b9: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-2/wal/wals/87b9ff3902364b85a0de9c8dc5d305e5
I20260928 22:51:39.314992 31656 ts_tablet_manager.cc:1939] T 87b9ff3902364b85a0de9c8dc5d305e5 P 87072ae366cd40b48beee55f28afb7b9: Deleting consensus metadata
I20260928 22:51:39.315609 31153 catalog_manager.cc:4841] TS 87072ae366cd40b48beee55f28afb7b9 (127.16.121.131:39589): tablet 87b9ff3902364b85a0de9c8dc5d305e5 (table test-table [id=20a3d436d145449d84e79c7195e4db61]) successfully deleted
W20260928 22:51:39.318424 31154 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.121.129:36367 (error 108)
I20260928 22:51:39.318949 31633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:39.318815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:36367 (local address 127.0.0.1:58966)
0928 22:51:39.318884 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:39.318921 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.129:36367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 22:51:39.318941 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 31360
W20260928 22:51:39.321483 31154 catalog_manager.cc:4584] TS 86d13097cb5348d2a58f4c202689f82e (127.16.121.129:36367): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 87b9ff3902364b85a0de9c8dc5d305e5: Network error: Client connection negotiation failed: client connection to 127.16.121.129:36367: connect: Connection refused (error 111)
I20260928 22:51:39.325681 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 31497
I20260928 22:51:39.330345 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 31134
I20260928 22:51:39.335932 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:37803
--webserver_interface=127.16.121.190
--webserver_port=39081
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:36021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:37803
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:51:39.419937 31658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:51:39.420156 31658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:51:39.420181 31658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:51:39.421307 31658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:51:39.421344 31658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:51:39.421358 31658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:51:39.421371 31658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:51:39.421382 31658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:51:39.422763 31658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:36021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:37803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:37803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.121.190
--webserver_port=39081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:51:39.422960 31658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:51:39.423156 31658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:51:39.427995 31663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:51:39.428058 31664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:51:39.428301 31658 server_base.cc:1034] running on GCE node
W20260928 22:51:39.428030 31666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:51:39.428610 31658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:51:39.428869 31658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:51:39.430011 31658 hybrid_clock.cc:648] HybridClock initialized: now 1790635899429995 us; error 28 us; skew 500 ppm
Sep 28 22:51:39 dist-test-slave-56vw krb5kdc[30586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635899, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:51:39.432642 31658 init.cc:377] Logged in from keytab as kudu/127.16.121.190@KRBTEST.COM (short username kudu)
I20260928 22:51:39.433087 31658 webserver.cc:460] Webserver started at http://127.16.121.190:39081/ using document root <none> and password file <none>
I20260928 22:51:39.433266 31658 fs_manager.cc:359] Metadata directory not provided
I20260928 22:51:39.433307 31658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:51:39.434566 31658 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:51:39.435223 31673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:39.435401 31658 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:51:39.435494 31658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "2fd24b851da0411193bead83d7411447"
format_stamp: "Formatted at 2026-09-28 22:51:37 on dist-test-slave-56vw"
server_key: "93823709de34f9f37ec9d781eb76d7b7"
server_key_iv: "a281b7f64ec9c6a39a0ca05d6f29e18b"
server_key_version: "encryptionkey@0"
I20260928 22:51:39.435832 31658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:51:39.452185 31658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:51:39.454610 31658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:51:39.454739 31658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:51:39.458704 31658 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.190:37803
I20260928 22:51:39.458783 31725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.190:37803 every 8 connection(s)
I20260928 22:51:39.459174 31658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:51:39.460678 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 31658
I20260928 22:51:39.460893 31726 sys_catalog.cc:263] Verifying existing consensus state
I20260928 22:51:39.461086 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.129:36367
--local_ip_for_outbound_sockets=127.16.121.129
--tserver_master_addrs=127.16.121.190:37803
--webserver_port=46821
--webserver_interface=127.16.121.129
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab
--principal=kudu/127.16.121.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:36021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:51:39.461715 31726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447: Bootstrap starting.
I20260928 22:51:39.470656 31726 log.cc:824] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447: Log is configured to *not* fsync() on all Append() calls
I20260928 22:51:39.473125 31726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 22:51:39.473470 31726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447: Bootstrap complete.
I20260928 22:51:39.475971 31726 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fd24b851da0411193bead83d7411447" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37803 } }
I20260928 22:51:39.476233 31726 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fd24b851da0411193bead83d7411447, State: Initialized, Role: FOLLOWER
I20260928 22:51:39.476447 31726 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fd24b851da0411193bead83d7411447" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37803 } }
I20260928 22:51:39.476545 31726 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:51:39.476600 31726 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:51:39.476649 31726 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [term 2 FOLLOWER]: Advancing to term 3
I20260928 22:51:39.477677 31726 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fd24b851da0411193bead83d7411447" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37803 } }
I20260928 22:51:39.477799 31726 leader_election.cc:302] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fd24b851da0411193bead83d7411447; no voters:
I20260928 22:51:39.478024 31726 leader_election.cc:288] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 22:51:39.478068 31730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 22:51:39.478209 31730 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [term 3 LEADER]: Becoming Leader. State: Replica: 2fd24b851da0411193bead83d7411447, State: Running, Role: LEADER
I20260928 22:51:39.478307 31726 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:51:39.478312 31730 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fd24b851da0411193bead83d7411447" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37803 } }
I20260928 22:51:39.478632 31730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fd24b851da0411193bead83d7411447. Latest consensus state: current_term: 3 leader_uuid: "2fd24b851da0411193bead83d7411447" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fd24b851da0411193bead83d7411447" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37803 } } }
I20260928 22:51:39.478641 31731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2fd24b851da0411193bead83d7411447" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fd24b851da0411193bead83d7411447" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37803 } } }
I20260928 22:51:39.478715 31730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [sys.catalog]: This master's current role is: LEADER
I20260928 22:51:39.478770 31731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fd24b851da0411193bead83d7411447 [sys.catalog]: This master's current role is: LEADER
I20260928 22:51:39.479255 31739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:51:39.480139 31739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:51:39.480294 31739 catalog_manager.cc:1259] Loaded cluster ID: a02d1fff1901438a888b8cd9d3c64119
I20260928 22:51:39.480499 31739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 22:51:39.480748 31739 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f02dcdaf9ad google::LogMessage::Fail() at ??:0
@ 0x7f02dcdb288a google::LogMessage::SendToLog() at ??:0
@ 0x7f02dcdaf4de google::LogMessage::Flush() at ??:0
@ 0x7f02dcdb0a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f02e4cd00f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f02e4cd98b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f02dec7d0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f02dec72954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f02dfad66db start_thread at ??:0
@ 0x7f02db8ad61f clone at ??:0
W20260928 22:51:39.583387 31728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:51:39.583683 31728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:51:39.583724 31728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:51:39.585631 31728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:51:39.585706 31728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:51:39.585755 31728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.129
I20260928 22:51:39.588092 31728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:36021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab
--principal=kudu/127.16.121.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.129:36367
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.121.129
--webserver_port=46821
--tserver_master_addrs=127.16.121.190:37803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.129
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:51:39.588380 31728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:51:39.588683 31728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:51:39.594501 31762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:51:39.594571 31765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:51:39.594791 31763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:51:39.594798 31728 server_base.cc:1034] running on GCE node
I20260928 22:51:39.595134 31728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:51:39.595431 31728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:51:39.596621 31728 hybrid_clock.cc:648] HybridClock initialized: now 1790635899596599 us; error 37 us; skew 500 ppm
Sep 28 22:51:39 dist-test-slave-56vw krb5kdc[30586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635899, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:51:39.599419 31728 init.cc:377] Logged in from keytab as kudu/127.16.121.129@KRBTEST.COM (short username kudu)
I20260928 22:51:39.599938 31728 webserver.cc:460] Webserver started at http://127.16.121.129:46821/ using document root <none> and password file <none>
I20260928 22:51:39.600183 31728 fs_manager.cc:359] Metadata directory not provided
I20260928 22:51:39.600253 31728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:51:39.601733 31728 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:51:39.602648 31772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:39.602877 31728 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:51:39.602981 31728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-0/wal
uuid: "86d13097cb5348d2a58f4c202689f82e"
format_stamp: "Formatted at 2026-09-28 22:51:37 on dist-test-slave-56vw"
server_key: "d1ab9569fc2b4e447e0579f94ac0eb03"
server_key_iv: "06a6de36040dba02659a2d0aa34e60a8"
server_key_version: "encryptionkey@0"
I20260928 22:51:39.603490 31728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:51:39.624794 31728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:51:39.627310 31728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:51:39.627482 31728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:51:39.627743 31728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:51:39.628269 31779 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 22:51:39.630167 31728 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 22:51:39.630232 31728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 22:51:39.630260 31728 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 22:51:39.631158 31728 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 22:51:39.631210 31728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.001s
I20260928 22:51:39.631289 31779 tablet_bootstrap.cc:492] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e: Bootstrap starting.
I20260928 22:51:39.632586 31779 log.cc:824] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e: Log is configured to *not* fsync() on all Append() calls
I20260928 22:51:39.635780 31779 tablet_bootstrap.cc:492] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 22:51:39.636111 31779 tablet_bootstrap.cc:492] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e: Bootstrap complete.
I20260928 22:51:39.636276 31779 ts_tablet_manager.cc:1397] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 22:51:39.638298 31779 raft_consensus.cc:348] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90486aa702d140f19e71be00c5088990" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 41643 } } peers { permanent_uuid: "87072ae366cd40b48beee55f28afb7b9" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 39589 } } peers { permanent_uuid: "86d13097cb5348d2a58f4c202689f82e" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 36367 } }
I20260928 22:51:39.638689 31779 raft_consensus.cc:729] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86d13097cb5348d2a58f4c202689f82e, State: Initialized, Role: FOLLOWER
I20260928 22:51:39.638929 31779 consensus_queue.cc:260] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90486aa702d140f19e71be00c5088990" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 41643 } } peers { permanent_uuid: "87072ae366cd40b48beee55f28afb7b9" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 39589 } } peers { permanent_uuid: "86d13097cb5348d2a58f4c202689f82e" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 36367 } }
I20260928 22:51:39.639249 31779 ts_tablet_manager.cc:1428] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:51:39.639707 31728 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.129:36367
I20260928 22:51:39.639737 31888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.129:36367 every 8 connection(s)
I20260928 22:51:39.640107 31728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
I20260928 22:51:39.647301 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 31728
I20260928 22:51:39.647603 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.130:41643
--local_ip_for_outbound_sockets=127.16.121.130
--tserver_master_addrs=127.16.121.190:37803
--webserver_port=42105
--webserver_interface=127.16.121.130
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab
--principal=kudu/127.16.121.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:36021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:51:39.730614 31892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:51:39.730898 31892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:51:39.730939 31892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:51:39.732236 31892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:51:39.732304 31892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:51:39.732350 31892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.130
I20260928 22:51:39.733825 31892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:36021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab
--principal=kudu/127.16.121.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.130:41643
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.121.130
--webserver_port=42105
--tserver_master_addrs=127.16.121.190:37803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.130
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:51:39.734102 31892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:51:39.734344 31892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:51:39.739970 31900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:51:39.740028 31891 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:51:39.640764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.129:47515)
0928 22:51:39.640938 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:39.640984 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 22:51:39.641145 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:51:39.739707 (+ 98562us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":69,"threads_started":1}
W20260928 22:51:39.740021 31898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:51:39.740226 31897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:51:39.740348 31889 heartbeater.cc:644] Failed to heartbeat to 127.16.121.190:37803 (0 consecutive failures): Network error: Failed to ping master at 127.16.121.190:37803: Client connection negotiation failed: client connection to 127.16.121.190:37803: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 22:51:39.740794 31891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:39.740628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.129:39575)
0928 22:51:39.740690 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:39.740759 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 22:51:39.740767 31892 server_base.cc:1034] running on GCE node
I20260928 22:51:39.741034 31892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:51:39.741225 31891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:39.741137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.129:34527)
0928 22:51:39.741174 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:39.741199 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260928 22:51:39.741343 31889 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 22:51:39.741370 31892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:51:39.742554 31892 hybrid_clock.cc:648] HybridClock initialized: now 1790635899742538 us; error 28 us; skew 500 ppm
Sep 28 22:51:39 dist-test-slave-56vw krb5kdc[30586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635899, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:51:39.745100 31892 init.cc:377] Logged in from keytab as kudu/127.16.121.130@KRBTEST.COM (short username kudu)
I20260928 22:51:39.745527 31892 webserver.cc:460] Webserver started at http://127.16.121.130:42105/ using document root <none> and password file <none>
I20260928 22:51:39.745709 31892 fs_manager.cc:359] Metadata directory not provided
I20260928 22:51:39.745748 31892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:51:39.747119 31892 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:51:39.747692 31907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:39.747844 31892 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:39.747936 31892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-1/wal
uuid: "90486aa702d140f19e71be00c5088990"
format_stamp: "Formatted at 2026-09-28 22:51:38 on dist-test-slave-56vw"
server_key: "c5b4f7ce9c47173fe00ad0a32ca73011"
server_key_iv: "627c6ae39d4c7a429c96adddce7d38cd"
server_key_version: "encryptionkey@0"
I20260928 22:51:39.748291 31892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:51:39.771502 31892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:51:39.774556 31892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:51:39.774717 31892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:51:39.774976 31892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:51:39.775324 31892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:51:39.775381 31892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:39.775426 31892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:51:39.775454 31892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:39.781745 31892 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.130:41643
I20260928 22:51:39.781785 32020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.130:41643 every 8 connection(s)
I20260928 22:51:39.782195 31892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
I20260928 22:51:39.783269 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 31892
I20260928 22:51:39.783489 32023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:39.782965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.130:49801)
0928 22:51:39.783175 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:39.783284 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":113,"threads_started":1}
I20260928 22:51:39.783587 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.131:39589
--local_ip_for_outbound_sockets=127.16.121.131
--tserver_master_addrs=127.16.121.190:37803
--webserver_port=45171
--webserver_interface=127.16.121.131
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab
--principal=kudu/127.16.121.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:36021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:51:39.785010 32021 heartbeater.cc:644] Failed to heartbeat to 127.16.121.190:37803 (0 consecutive failures): Network error: Failed to ping master at 127.16.121.190:37803: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
I20260928 22:51:39.785331 32023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:39.785219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.130:47541)
0928 22:51:39.785274 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:39.785310 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 22:51:39.785610 32023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:39.785521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.130:38451)
0928 22:51:39.785565 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:39.785587 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 22:51:39.785754 32021 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 22:51:39.867304 32024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:51:39.867527 32024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:51:39.867552 32024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:51:39.868786 32024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:51:39.868840 32024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:51:39.868867 32024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.131
I20260928 22:51:39.870306 32024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:36021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab
--principal=kudu/127.16.121.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.131:39589
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.121.131
--webserver_port=45171
--tserver_master_addrs=127.16.121.190:37803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.131
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:51:39.870507 32024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:51:39.870690 32024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:51:39.876096 32032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:51:39.876086 32030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:51:39.876147 32024 server_base.cc:1034] running on GCE node
W20260928 22:51:39.876088 32029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:51:39.876525 32024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:51:39.876794 32024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:51:39.877950 32024 hybrid_clock.cc:648] HybridClock initialized: now 1790635899877941 us; error 50 us; skew 500 ppm
Sep 28 22:51:39 dist-test-slave-56vw krb5kdc[30586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635899, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:51:39.880715 32024 init.cc:377] Logged in from keytab as kudu/127.16.121.131@KRBTEST.COM (short username kudu)
I20260928 22:51:39.881356 32024 webserver.cc:460] Webserver started at http://127.16.121.131:45171/ using document root <none> and password file <none>
I20260928 22:51:39.881659 32024 fs_manager.cc:359] Metadata directory not provided
I20260928 22:51:39.881744 32024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:51:39.883361 32024 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:51:39.884194 32039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:39.884433 32024 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:51:39.884509 32024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-2/wal
uuid: "87072ae366cd40b48beee55f28afb7b9"
format_stamp: "Formatted at 2026-09-28 22:51:38 on dist-test-slave-56vw"
server_key: "731b56b30449e31d337fb18fa7a0f486"
server_key_iv: "840d244dc374d7f4e039b7ed8a04cc69"
server_key_version: "encryptionkey@0"
I20260928 22:51:39.884920 32024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:51:39.896092 32024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:51:39.898216 32024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:51:39.898375 32024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:51:39.898660 32024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:51:39.898989 32024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:51:39.899046 32024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:39.899152 32024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:51:39.899183 32024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:39.905359 32024 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.131:39589
I20260928 22:51:39.905406 32152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.131:39589 every 8 connection(s)
I20260928 22:51:39.905819 32024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
I20260928 22:51:39.907021 32155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:39.906529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.131:36349)
0928 22:51:39.906696 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:39.906864 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":63,"threads_started":1}
W20260928 22:51:39.907442 32153 heartbeater.cc:644] Failed to heartbeat to 127.16.121.190:37803 (0 consecutive failures): Network error: Failed to ping master at 127.16.121.190:37803: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
I20260928 22:51:39.907716 32155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:39.907624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.131:50541)
0928 22:51:39.907672 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:39.907699 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 22:51:39.907945 32155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:39.907897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.131:38619)
0928 22:51:39.907917 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:39.907934 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260928 22:51:39.908052 32153 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 22:51:39.908643 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 32024
I20260928 22:51:40.742003 32157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:40.741757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.129:47487)
0928 22:51:40.741901 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:40.741937 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":85,"threads_started":1}
I20260928 22:51:40.786350 32158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:40.786102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.130:36693)
0928 22:51:40.786257 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:40.786293 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":60,"threads_started":1}
I20260928 22:51:40.908708 32159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:40.908466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.131:53591)
0928 22:51:40.908586 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:40.908654 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
I20260928 22:51:41.312273 32160 raft_consensus.cc:482] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:51:41.312413 32160 raft_consensus.cc:504] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90486aa702d140f19e71be00c5088990" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 41643 } } peers { permanent_uuid: "87072ae366cd40b48beee55f28afb7b9" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 39589 } } peers { permanent_uuid: "86d13097cb5348d2a58f4c202689f82e" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 36367 } }
I20260928 22:51:41.312750 32160 leader_election.cc:288] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 90486aa702d140f19e71be00c5088990 (127.16.121.130:41643), 87072ae366cd40b48beee55f28afb7b9 (127.16.121.131:39589)
Sep 28 22:51:41 dist-test-slave-56vw krb5kdc[30586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635899, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for kudu/127.16.121.131@KRBTEST.COM
Sep 28 22:51:41 dist-test-slave-56vw krb5kdc[30586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635899, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for kudu/127.16.121.130@KRBTEST.COM
I20260928 22:51:41.331586 32163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:41.313147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:53233 (local address 127.16.121.130:41643)
0928 22:51:41.313317 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:41.313326 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:41.313346 (+ 20us) server_negotiation.cc:407] Connection header received
0928 22:51:41.313769 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:41.313772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:41.313812 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:41.313963 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:51:41.314761 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:41.315701 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:41.316884 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:41.317285 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:41.317408 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:41.328077 (+ 10669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:51:41.328082 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:51:41.328082 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:51:41.328094 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:51:41.329752 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:41.330274 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:41.330277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:41.330277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:41.330381 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:41.330645 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:41.330646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:41.330647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:41.330901 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:51:41.330991 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:41.331324 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 22:51:41.331468 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":53,"threads_started":1}
I20260928 22:51:41.331586 32161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:41.312997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.130:41643 (local address 127.16.121.129:53233)
0928 22:51:41.313152 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:41.313165 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:51:41.313227 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:51:41.314005 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:51:41.314008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:51:41.314401 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:51:41.314626 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:41.314632 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:41.315808 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:41.315809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:41.316782 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:41.316784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:41.317385 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:41.317386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:41.317516 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:41.324977 (+ 7461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:51:41.324993 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:51:41.327927 (+ 2934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:51:41.329872 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:41.329873 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:41.329873 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:41.330184 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:41.330463 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:41.330464 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:41.330464 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:41.330577 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:41.330984 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:51:41.330990 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:51:41.331160 (+ 170us) client_negotiation.cc:769] Sending connection context
0928 22:51:41.331271 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 22:51:41.331475 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":19,"thread_start_us":80,"threads_started":1}
I20260928 22:51:41.331732 32162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:41.313021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.131:39589 (local address 127.16.121.129:57341)
0928 22:51:41.313170 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:41.313181 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:51:41.313238 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:51:41.314019 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:51:41.314021 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:51:41.314411 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:51:41.314815 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:41.314818 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:41.316037 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:41.316038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:41.316804 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:41.316807 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:41.317456 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:41.317457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:41.317525 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:41.324990 (+ 7465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:51:41.324996 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:51:41.327340 (+ 2344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:51:41.329834 (+ 2494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:41.329842 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:41.329844 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:41.330199 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:41.330496 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:41.330497 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:41.330498 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:41.330584 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:41.331001 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:51:41.331002 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:51:41.331160 (+ 158us) client_negotiation.cc:769] Sending connection context
0928 22:51:41.331271 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 22:51:41.331572 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":142,"thread_start_us":64,"threads_started":1}
I20260928 22:51:41.331642 32164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:41.313133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:57341 (local address 127.16.121.131:39589)
0928 22:51:41.313329 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:41.313348 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:41.313377 (+ 29us) server_negotiation.cc:407] Connection header received
0928 22:51:41.313821 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:41.313824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:41.313862 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:41.314028 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:51:41.314948 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:41.315936 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:41.316898 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:41.317353 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:41.317490 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:41.327511 (+ 10021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:51:41.327517 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:51:41.327517 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:51:41.327533 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:51:41.329728 (+ 2195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:41.330301 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:41.330304 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:41.330305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:41.330401 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:41.330666 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:41.330666 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:41.330667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:41.330920 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:51:41.331019 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:41.331342 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 22:51:41.331514 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":76,"threads_started":1}
W20260928 22:51:41.332787 31774 leader_election.cc:341] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 90486aa702d140f19e71be00c5088990 (127.16.121.130:41643): Not found: Tablet not found: 87b9ff3902364b85a0de9c8dc5d305e5
W20260928 22:51:41.332928 31775 leader_election.cc:341] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87072ae366cd40b48beee55f28afb7b9 (127.16.121.131:39589): Not found: Tablet not found: 87b9ff3902364b85a0de9c8dc5d305e5
I20260928 22:51:41.333009 31775 leader_election.cc:302] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 86d13097cb5348d2a58f4c202689f82e; no voters: 87072ae366cd40b48beee55f28afb7b9, 90486aa702d140f19e71be00c5088990
I20260928 22:51:41.333110 32160 raft_consensus.cc:2726] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 22:51:41.742673 32162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:41.742536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.129:44703)
0928 22:51:41.742593 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:41.742630 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 22:51:41.787321 32165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:41.786901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.130:46153)
0928 22:51:41.787036 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:41.787256 (+ 220us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":64,"threads_started":1}
I20260928 22:51:41.909578 32166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:41.909328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.131:56069)
0928 22:51:41.909488 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:41.909534 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":65,"threads_started":1}
I20260928 22:51:42.743558 32167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:42.743300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.129:55651)
0928 22:51:42.743468 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:42.743510 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":65,"threads_started":1}
I20260928 22:51:42.788100 32168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:42.787835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.130:52637)
0928 22:51:42.787998 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:42.788042 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":76,"threads_started":1}
I20260928 22:51:42.910394 32169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:42.910153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.131:43141)
0928 22:51:42.910304 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:42.910350 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":90,"threads_started":1}
I20260928 22:51:43.174273 32170 raft_consensus.cc:482] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:51:43.174392 32170 raft_consensus.cc:504] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90486aa702d140f19e71be00c5088990" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 41643 } } peers { permanent_uuid: "87072ae366cd40b48beee55f28afb7b9" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 39589 } } peers { permanent_uuid: "86d13097cb5348d2a58f4c202689f82e" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 36367 } }
I20260928 22:51:43.174609 32170 leader_election.cc:288] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 90486aa702d140f19e71be00c5088990 (127.16.121.130:41643), 87072ae366cd40b48beee55f28afb7b9 (127.16.121.131:39589)
W20260928 22:51:43.175182 31775 leader_election.cc:341] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87072ae366cd40b48beee55f28afb7b9 (127.16.121.131:39589): Not found: Tablet not found: 87b9ff3902364b85a0de9c8dc5d305e5
W20260928 22:51:43.175309 31774 leader_election.cc:341] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 90486aa702d140f19e71be00c5088990 (127.16.121.130:41643): Not found: Tablet not found: 87b9ff3902364b85a0de9c8dc5d305e5
I20260928 22:51:43.175385 31774 leader_election.cc:302] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 86d13097cb5348d2a58f4c202689f82e; no voters: 87072ae366cd40b48beee55f28afb7b9, 90486aa702d140f19e71be00c5088990
I20260928 22:51:43.175454 32170 raft_consensus.cc:2726] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 22:51:43.744475 32171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:43.744226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.129:50717)
0928 22:51:43.744383 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:43.744427 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":109,"threads_started":1}
I20260928 22:51:43.788875 32172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:43.788596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.130:57137)
0928 22:51:43.788769 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:43.788809 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":74,"threads_started":1}
I20260928 22:51:43.911419 32173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:43.911128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.131:37325)
0928 22:51:43.911320 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:43.911369 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":74,"threads_started":1}
I20260928 22:51:44.745345 32174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:44.745049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.129:53539)
0928 22:51:44.745248 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:44.745296 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":88,"threads_started":1}
I20260928 22:51:44.789754 32175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:44.789507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.130:40949)
0928 22:51:44.789659 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:44.789708 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":67,"threads_started":1}
I20260928 22:51:44.912613 32176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:44.912278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.131:37167)
0928 22:51:44.912508 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:44.912563 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":77,"threads_started":1}
I20260928 22:51:45.468430 32177 raft_consensus.cc:482] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:51:45.468540 32177 raft_consensus.cc:504] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90486aa702d140f19e71be00c5088990" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 41643 } } peers { permanent_uuid: "87072ae366cd40b48beee55f28afb7b9" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 39589 } } peers { permanent_uuid: "86d13097cb5348d2a58f4c202689f82e" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 36367 } }
I20260928 22:51:45.468777 32177 leader_election.cc:288] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 90486aa702d140f19e71be00c5088990 (127.16.121.130:41643), 87072ae366cd40b48beee55f28afb7b9 (127.16.121.131:39589)
W20260928 22:51:45.469221 31775 leader_election.cc:341] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87072ae366cd40b48beee55f28afb7b9 (127.16.121.131:39589): Not found: Tablet not found: 87b9ff3902364b85a0de9c8dc5d305e5
W20260928 22:51:45.469404 31774 leader_election.cc:341] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 90486aa702d140f19e71be00c5088990 (127.16.121.130:41643): Not found: Tablet not found: 87b9ff3902364b85a0de9c8dc5d305e5
I20260928 22:51:45.469484 31774 leader_election.cc:302] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 86d13097cb5348d2a58f4c202689f82e; no voters: 87072ae366cd40b48beee55f28afb7b9, 90486aa702d140f19e71be00c5088990
I20260928 22:51:45.469585 32177 raft_consensus.cc:2726] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 22:51:45.746420 32178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:45.746166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.129:40267)
0928 22:51:45.746315 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:45.746370 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":68,"threads_started":1}
I20260928 22:51:45.790637 32179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:45.790345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.130:44213)
0928 22:51:45.790530 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:45.790581 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":74,"threads_started":1}
I20260928 22:51:45.913610 32180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:45.913318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.131:60383)
0928 22:51:45.913509 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:45.913560 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":70,"threads_started":1}
I20260928 22:51:46.747344 32181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:46.747025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.129:51083)
0928 22:51:46.747249 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:46.747295 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":106,"threads_started":1}
I20260928 22:51:46.791574 32182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:46.791292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.130:44117)
0928 22:51:46.791469 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:46.791515 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":81,"threads_started":1}
I20260928 22:51:46.914630 32183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:46.914286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.131:35767)
0928 22:51:46.914453 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:46.914501 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":69,"threads_started":1}
I20260928 22:51:47.748155 32184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:47.747907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.129:60639)
0928 22:51:47.748063 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:47.748101 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":79,"threads_started":1}
I20260928 22:51:47.792428 32185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:47.792188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.130:40151)
0928 22:51:47.792329 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:47.792380 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":71,"threads_started":1}
I20260928 22:51:47.915670 32186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:47.915408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.131:58507)
0928 22:51:47.915564 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:47.915618 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":65,"threads_started":1}
I20260928 22:51:48.748996 32187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:48.748773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.129:38487)
0928 22:51:48.748901 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:48.748943 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":65,"threads_started":1}
I20260928 22:51:48.793473 32188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:48.793162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.130:57477)
0928 22:51:48.793362 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:48.793425 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":74,"threads_started":1}
I20260928 22:51:48.846921 32189 raft_consensus.cc:482] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:51:48.847107 32189 raft_consensus.cc:504] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90486aa702d140f19e71be00c5088990" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 41643 } } peers { permanent_uuid: "87072ae366cd40b48beee55f28afb7b9" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 39589 } } peers { permanent_uuid: "86d13097cb5348d2a58f4c202689f82e" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 36367 } }
I20260928 22:51:48.847393 32189 leader_election.cc:288] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 90486aa702d140f19e71be00c5088990 (127.16.121.130:41643), 87072ae366cd40b48beee55f28afb7b9 (127.16.121.131:39589)
W20260928 22:51:48.847831 31774 leader_election.cc:341] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 90486aa702d140f19e71be00c5088990 (127.16.121.130:41643): Not found: Tablet not found: 87b9ff3902364b85a0de9c8dc5d305e5
W20260928 22:51:48.847982 31775 leader_election.cc:341] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87072ae366cd40b48beee55f28afb7b9 (127.16.121.131:39589): Not found: Tablet not found: 87b9ff3902364b85a0de9c8dc5d305e5
I20260928 22:51:48.848040 31775 leader_election.cc:302] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 86d13097cb5348d2a58f4c202689f82e; no voters: 87072ae366cd40b48beee55f28afb7b9, 90486aa702d140f19e71be00c5088990
I20260928 22:51:48.848114 32189 raft_consensus.cc:2726] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 22:51:48.916728 32190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:48.916445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.131:37893)
0928 22:51:48.916614 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:48.916660 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":108,"threads_started":1}
I20260928 22:51:49.749846 32191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:49.749616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.129:43287)
0928 22:51:49.749753 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:49.749797 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":70,"threads_started":1}
I20260928 22:51:49.794627 32192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:49.794335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.130:59605)
0928 22:51:49.794484 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:49.794540 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":93,"threads_started":1}
I20260928 22:51:49.917647 32193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:49.917371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.131:40015)
0928 22:51:49.917538 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:49.917596 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":63,"threads_started":1}
I20260928 22:51:50.698732 32194 raft_consensus.cc:482] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:51:50.698834 32194 raft_consensus.cc:504] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90486aa702d140f19e71be00c5088990" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 41643 } } peers { permanent_uuid: "87072ae366cd40b48beee55f28afb7b9" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 39589 } } peers { permanent_uuid: "86d13097cb5348d2a58f4c202689f82e" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 36367 } }
I20260928 22:51:50.699048 32194 leader_election.cc:288] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 90486aa702d140f19e71be00c5088990 (127.16.121.130:41643), 87072ae366cd40b48beee55f28afb7b9 (127.16.121.131:39589)
W20260928 22:51:50.699501 31774 leader_election.cc:341] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 90486aa702d140f19e71be00c5088990 (127.16.121.130:41643): Not found: Tablet not found: 87b9ff3902364b85a0de9c8dc5d305e5
W20260928 22:51:50.699575 31775 leader_election.cc:341] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87072ae366cd40b48beee55f28afb7b9 (127.16.121.131:39589): Not found: Tablet not found: 87b9ff3902364b85a0de9c8dc5d305e5
I20260928 22:51:50.699617 31774 leader_election.cc:302] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 86d13097cb5348d2a58f4c202689f82e; no voters: 87072ae366cd40b48beee55f28afb7b9, 90486aa702d140f19e71be00c5088990
I20260928 22:51:50.699699 32194 raft_consensus.cc:2726] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 22:51:50.750837 32195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:50.750564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.129:47667)
0928 22:51:50.750745 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:50.750791 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":127,"threads_started":1}
I20260928 22:51:50.795506 32196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:50.795271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.130:34873)
0928 22:51:50.795405 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:50.795453 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":63,"threads_started":1}
I20260928 22:51:50.918462 32197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:50.918214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.131:53213)
0928 22:51:50.918363 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:50.918412 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":96,"threads_started":1}
I20260928 22:51:51.751727 32198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:51.751413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.129:55597)
0928 22:51:51.751637 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:51.751678 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":73,"threads_started":1}
I20260928 22:51:51.796281 32199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:51.795990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.130:44977)
0928 22:51:51.796180 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:51.796233 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":82,"threads_started":1}
I20260928 22:51:51.919443 32200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:51.919107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.131:51007)
0928 22:51:51.919325 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:51.919380 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":125,"threads_started":1}
I20260928 22:51:52.752506 32201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:52.752272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.129:56349)
0928 22:51:52.752413 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:52.752456 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":73,"threads_started":1}
I20260928 22:51:52.797252 32202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:52.796949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.130:51923)
0928 22:51:52.797118 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:52.797187 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":83,"threads_started":1}
I20260928 22:51:52.920451 32203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:52.920151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.131:52709)
0928 22:51:52.920352 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:52.920396 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":72,"threads_started":1}
I20260928 22:51:53.753423 32204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:53.753139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.129:59593)
0928 22:51:53.753296 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:53.753340 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":96,"threads_started":1}
I20260928 22:51:53.798055 32205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:53.797821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.130:43213)
0928 22:51:53.797965 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:53.798008 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":72,"threads_started":1}
I20260928 22:51:53.921386 32206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:53.921082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.131:34645)
0928 22:51:53.921287 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:53.921335 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":67,"threads_started":1}
I20260928 22:51:54.009858 32207 raft_consensus.cc:482] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:51:54.010093 32207 raft_consensus.cc:504] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90486aa702d140f19e71be00c5088990" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 41643 } } peers { permanent_uuid: "87072ae366cd40b48beee55f28afb7b9" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 39589 } } peers { permanent_uuid: "86d13097cb5348d2a58f4c202689f82e" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 36367 } }
I20260928 22:51:54.010362 32207 leader_election.cc:288] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 90486aa702d140f19e71be00c5088990 (127.16.121.130:41643), 87072ae366cd40b48beee55f28afb7b9 (127.16.121.131:39589)
W20260928 22:51:54.010756 31774 leader_election.cc:341] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 90486aa702d140f19e71be00c5088990 (127.16.121.130:41643): Not found: Tablet not found: 87b9ff3902364b85a0de9c8dc5d305e5
W20260928 22:51:54.010844 31775 leader_election.cc:341] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87072ae366cd40b48beee55f28afb7b9 (127.16.121.131:39589): Not found: Tablet not found: 87b9ff3902364b85a0de9c8dc5d305e5
I20260928 22:51:54.010908 31775 leader_election.cc:302] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 86d13097cb5348d2a58f4c202689f82e; no voters: 87072ae366cd40b48beee55f28afb7b9, 90486aa702d140f19e71be00c5088990
I20260928 22:51:54.010998 32207 raft_consensus.cc:2726] T 87b9ff3902364b85a0de9c8dc5d305e5 P 86d13097cb5348d2a58f4c202689f82e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 22:51:54.754456 32208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:54.754157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.129:53603)
0928 22:51:54.754345 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:54.754397 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":75,"threads_started":1}
I20260928 22:51:54.798947 32209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:54.798693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.130:55691)
0928 22:51:54.798854 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:54.798900 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":71,"threads_started":1}
I20260928 22:51:54.916806 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 31728
I20260928 22:51:54.922103 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 31892
I20260928 22:51:54.922259 32210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:54.922030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37803 (local address 127.16.121.131:43619)
0928 22:51:54.922159 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:54.922202 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":52,"threads_started":1}
I20260928 22:51:54.927016 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 32024
2026-09-28T22:51:54Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18444 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:51:55 dist-test-slave-56vw krb5kdc[32213](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:51:55 dist-test-slave-56vw krb5kdc[32213](info): set up 2 sockets
Sep 28 22:51:55 dist-test-slave-56vw krb5kdc[32213](info): commencing operation
krb5kdc: starting...
W20260928 22:51:56.011382 16870 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Sep 28 22:51:56 dist-test-slave-56vw krb5kdc[32213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:51:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:51:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.121.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
I20260928 22:51:56.141994 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:34005
--webserver_interface=127.16.121.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:42335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:34005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:51:56.225932 32226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:51:56.226202 32226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:51:56.226240 32226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:51:56.227530 32226 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 22:51:56.227602 32226 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 22:51:56.227623 32226 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 22:51:56.227643 32226 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:51:56.227674 32226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:51:56.227703 32226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:51:56.227731 32226 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:51:56.227758 32226 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:51:56.229348 32226 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:42335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:34005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:34005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.121.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:51:56.229642 32226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:51:56.229884 32226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:51:56.235013 32231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:51:56.235041 32234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:51:56.235107 32232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:51:56.235280 32226 server_base.cc:1034] running on GCE node
I20260928 22:51:56.235769 32226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:51:56.236091 32226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:51:56.237425 32226 hybrid_clock.cc:648] HybridClock initialized: now 1790635916237400 us; error 46 us; skew 500 ppm
Sep 28 22:51:56 dist-test-slave-56vw krb5kdc[32213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635916, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:51:56.240439 32226 init.cc:377] Logged in from keytab as kudu/127.16.121.190@KRBTEST.COM (short username kudu)
I20260928 22:51:56.241006 32226 webserver.cc:460] Webserver started at http://127.16.121.190:37533/ using document root <none> and password file <none>
I20260928 22:51:56.241294 32226 fs_manager.cc:359] Metadata directory not provided
I20260928 22:51:56.241366 32226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:51:56.241516 32226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:51:56.242727 32226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "6c9fdbbae88b4180bcbb87227139de8c"
format_stamp: "Formatted at 2026-09-28 22:51:56 on dist-test-slave-56vw"
server_key: "6f6ce0f70f5a723c4b3876801326714f"
server_key_iv: "db908605c2ebfc5e870579b9e8dd415f"
server_key_version: "encryptionkey@0"
I20260928 22:51:56.243163 32226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "6c9fdbbae88b4180bcbb87227139de8c"
format_stamp: "Formatted at 2026-09-28 22:51:56 on dist-test-slave-56vw"
server_key: "6f6ce0f70f5a723c4b3876801326714f"
server_key_iv: "db908605c2ebfc5e870579b9e8dd415f"
server_key_version: "encryptionkey@0"
I20260928 22:51:56.244628 32226 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:51:56.245560 32241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:56.245760 32226 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:51:56.245865 32226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "6c9fdbbae88b4180bcbb87227139de8c"
format_stamp: "Formatted at 2026-09-28 22:51:56 on dist-test-slave-56vw"
server_key: "6f6ce0f70f5a723c4b3876801326714f"
server_key_iv: "db908605c2ebfc5e870579b9e8dd415f"
server_key_version: "encryptionkey@0"
I20260928 22:51:56.245949 32226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:51:56.256520 32226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:51:56.258770 32226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:51:56.258944 32226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:51:56.263171 32226 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.190:34005
I20260928 22:51:56.263222 32293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.190:34005 every 8 connection(s)
I20260928 22:51:56.263614 32226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:51:56.264586 32294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:51:56.267002 32294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c: Bootstrap starting.
I20260928 22:51:56.267606 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 32226
I20260928 22:51:56.267714 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:51:56.267824 32294 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c: Neither blocks nor log segments found. Creating new log.
I20260928 22:51:56.267941 16870 external_mini_cluster.cc:1428] Setting key 4546cadd2570581661125caa390c5b65
I20260928 22:51:56.268164 32294 log.cc:824] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c: Log is configured to *not* fsync() on all Append() calls
I20260928 22:51:56.268970 32294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c: No bootstrap required, opened a new log
I20260928 22:51:56.271497 32294 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c9fdbbae88b4180bcbb87227139de8c" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 34005 } }
I20260928 22:51:56.271692 32294 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:51:56.271730 32294 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c9fdbbae88b4180bcbb87227139de8c, State: Initialized, Role: FOLLOWER
I20260928 22:51:56.271855 32294 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c9fdbbae88b4180bcbb87227139de8c" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 34005 } }
I20260928 22:51:56.271927 32294 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:51:56.271955 32294 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:51:56.271992 32294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:51:56.272665 32294 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c9fdbbae88b4180bcbb87227139de8c" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 34005 } }
I20260928 22:51:56.272790 32294 leader_election.cc:302] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c9fdbbae88b4180bcbb87227139de8c; no voters:
I20260928 22:51:56.273021 32294 leader_election.cc:288] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:51:56.273046 32299 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:51:56.273164 32299 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [term 1 LEADER]: Becoming Leader. State: Replica: 6c9fdbbae88b4180bcbb87227139de8c, State: Running, Role: LEADER
I20260928 22:51:56.273309 32299 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c9fdbbae88b4180bcbb87227139de8c" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 34005 } }
I20260928 22:51:56.273340 32294 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:51:56.273710 32301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c9fdbbae88b4180bcbb87227139de8c. Latest consensus state: current_term: 1 leader_uuid: "6c9fdbbae88b4180bcbb87227139de8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c9fdbbae88b4180bcbb87227139de8c" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 34005 } } }
I20260928 22:51:56.273715 32300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c9fdbbae88b4180bcbb87227139de8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c9fdbbae88b4180bcbb87227139de8c" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 34005 } } }
I20260928 22:51:56.273864 32301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [sys.catalog]: This master's current role is: LEADER
I20260928 22:51:56.273872 32300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [sys.catalog]: This master's current role is: LEADER
I20260928 22:51:56.274175 32304 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:51:56.274727 32304 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 22:51:56 dist-test-slave-56vw krb5kdc[32213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:51:56.276767 32304 catalog_manager.cc:1347] Generated new cluster ID: c471a56f6b4143c9b6e4cfa2a6484473
I20260928 22:51:56.276830 32304 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:51:56.279397 32298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:56.268578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52906 (local address 127.16.121.190:34005)
0928 22:51:56.268748 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:56.268758 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:56.268792 (+ 34us) server_negotiation.cc:407] Connection header received
0928 22:51:56.269279 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:56.269286 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:56.269336 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:56.269456 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:51:56.270238 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:56.271228 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:56.272295 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:56.272714 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:56.272875 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:56.276002 (+ 3127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:51:56.276006 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:51:56.276007 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:51:56.276024 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:51:56.277819 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:56.278208 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:56.278211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:56.278211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:56.278321 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:56.278499 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:56.278500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:56.278500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:56.278765 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:51:56.278882 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:56.278995 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 22:51:56.279165 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":73,"threads_started":1}
I20260928 22:51:56.303093 32304 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:51:56.303938 32304 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:51:56.306798 32304 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c: Generated new TSK 0
I20260928 22:51:56.307016 32304 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.121.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab.
Entry for principal kudu/127.16.121.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab.
I20260928 22:51:56.347335 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.129:0
--local_ip_for_outbound_sockets=127.16.121.129
--webserver_interface=127.16.121.129
--webserver_port=0
--tserver_master_addrs=127.16.121.190:34005
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab
--principal=kudu/127.16.121.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:42335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:51:56.441012 32320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:51:56.441258 32320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:51:56.441285 32320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:51:56.442544 32320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:51:56.442592 32320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:51:56.442621 32320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.129
I20260928 22:51:56.444284 32320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:42335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab
--principal=kudu/127.16.121.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.121.129
--webserver_port=0
--tserver_master_addrs=127.16.121.190:34005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.129
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:51:56.444563 32320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:51:56.444811 32320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:51:56.450029 32326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:51:56.450033 32325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:51:56.450343 32320 server_base.cc:1034] running on GCE node
W20260928 22:51:56.450052 32328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:51:56.450634 32320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:51:56.450883 32320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:51:56.452037 32320 hybrid_clock.cc:648] HybridClock initialized: now 1790635916452029 us; error 39 us; skew 500 ppm
Sep 28 22:51:56 dist-test-slave-56vw krb5kdc[32213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635916, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:51:56.454634 32320 init.cc:377] Logged in from keytab as kudu/127.16.121.129@KRBTEST.COM (short username kudu)
I20260928 22:51:56.455158 32320 webserver.cc:460] Webserver started at http://127.16.121.129:33913/ using document root <none> and password file <none>
I20260928 22:51:56.455413 32320 fs_manager.cc:359] Metadata directory not provided
I20260928 22:51:56.455472 32320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:51:56.455566 32320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:51:56.456605 32320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-0/data/instance:
uuid: "64000c9c5f484909b35f01e352a21041"
format_stamp: "Formatted at 2026-09-28 22:51:56 on dist-test-slave-56vw"
server_key: "3979c4e8dcb7661137b395086cd04247"
server_key_iv: "9010955251a31455e4f4fbb931c618df"
server_key_version: "encryptionkey@0"
I20260928 22:51:56.456933 32320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance:
uuid: "64000c9c5f484909b35f01e352a21041"
format_stamp: "Formatted at 2026-09-28 22:51:56 on dist-test-slave-56vw"
server_key: "3979c4e8dcb7661137b395086cd04247"
server_key_iv: "9010955251a31455e4f4fbb931c618df"
server_key_version: "encryptionkey@0"
I20260928 22:51:56.458277 32320 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:51:56.459107 32335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:56.459287 32320 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:51:56.459385 32320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-0/wal
uuid: "64000c9c5f484909b35f01e352a21041"
format_stamp: "Formatted at 2026-09-28 22:51:56 on dist-test-slave-56vw"
server_key: "3979c4e8dcb7661137b395086cd04247"
server_key_iv: "9010955251a31455e4f4fbb931c618df"
server_key_version: "encryptionkey@0"
I20260928 22:51:56.459474 32320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:51:56.475338 32320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:51:56.477805 32320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:51:56.478029 32320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:51:56.478331 32320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:51:56.478808 32320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:51:56.478865 32320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:56.478904 32320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:51:56.478935 32320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:56.485793 32320 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.129:37381
I20260928 22:51:56.485841 32448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.129:37381 every 8 connection(s)
I20260928 22:51:56.486200 32320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
Sep 28 22:51:56 dist-test-slave-56vw krb5kdc[32213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635916, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:51:56.493736 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 32320
I20260928 22:51:56.493914 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance
I20260928 22:51:56.494169 16870 external_mini_cluster.cc:1428] Setting key 1353eec2f69d4c3b1d99bf2246fa686d
I20260928 22:51:56.496528 32298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:56.486971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:48741 (local address 127.16.121.190:34005)
0928 22:51:56.487056 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:56.487059 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:56.487286 (+ 227us) server_negotiation.cc:407] Connection header received
0928 22:51:56.487867 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:56.487868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:56.487881 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:56.487946 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:51:56.489152 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:56.489668 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:56.490759 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:56.490983 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:56.491085 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:56.493224 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:51:56.493226 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:51:56.493226 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:51:56.493240 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:51:56.494773 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:56.495360 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:56.495362 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:56.495363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:56.495432 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:56.495721 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:56.495722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:56.495722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:56.495877 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:51:56.495939 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:56.496309 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 22:51:56.496421 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 22:51:56.496985 32451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:56.486927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.129:48741)
0928 22:51:56.487143 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:56.487186 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 22:51:56.487749 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:51:56.488024 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:51:56.488028 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:51:56.488817 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:51:56.489023 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:56.489029 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:56.489766 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:56.489767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:56.490640 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:56.490642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:56.491097 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:56.491098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:56.491234 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:56.491747 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:51:56.491755 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:51:56.493115 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:51:56.494887 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:56.494896 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:56.494898 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:56.495260 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:56.495511 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:56.495511 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:56.495512 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:56.495623 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:56.495980 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:51:56.495988 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:51:56.496141 (+ 153us) client_negotiation.cc:769] Sending connection context
0928 22:51:56.496235 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 22:51:56.496571 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":83,"threads_started":1}
I20260928 22:51:56.497615 32449 heartbeater.cc:347] Connected to a master server at 127.16.121.190:34005
I20260928 22:51:56.497735 32449 heartbeater.cc:464] Registering TS with master...
I20260928 22:51:56.497958 32449 heartbeater.cc:511] Master 127.16.121.190:34005 requested a full tablet report, sending...
I20260928 22:51:56.498544 32258 ts_manager.cc:194] Registered new tserver with Master: 64000c9c5f484909b35f01e352a21041 (127.16.121.129:37381)
I20260928 22:51:56.499590 32258 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.129@KRBTEST.COM'} at 127.16.121.129:48741
WARNING: no policy specified for kudu/127.16.121.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab.
Entry for principal kudu/127.16.121.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab.
I20260928 22:51:56.523002 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.130:0
--local_ip_for_outbound_sockets=127.16.121.130
--webserver_interface=127.16.121.130
--webserver_port=0
--tserver_master_addrs=127.16.121.190:34005
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab
--principal=kudu/127.16.121.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:42335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:51:56.607828 32454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:51:56.608054 32454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:51:56.608080 32454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:51:56.609238 32454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:51:56.609277 32454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:51:56.609303 32454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.130
I20260928 22:51:56.610707 32454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:42335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab
--principal=kudu/127.16.121.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.121.130
--webserver_port=0
--tserver_master_addrs=127.16.121.190:34005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.130
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:51:56.610903 32454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:51:56.611112 32454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:51:56.616209 32460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:51:56.616195 32462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:51:56.616195 32459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:51:56.616321 32454 server_base.cc:1034] running on GCE node
I20260928 22:51:56.616636 32454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:51:56.616886 32454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:51:56.618041 32454 hybrid_clock.cc:648] HybridClock initialized: now 1790635916618026 us; error 27 us; skew 500 ppm
Sep 28 22:51:56 dist-test-slave-56vw krb5kdc[32213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635916, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:51:56.620702 32454 init.cc:377] Logged in from keytab as kudu/127.16.121.130@KRBTEST.COM (short username kudu)
I20260928 22:51:56.621104 32454 webserver.cc:460] Webserver started at http://127.16.121.130:44219/ using document root <none> and password file <none>
I20260928 22:51:56.621296 32454 fs_manager.cc:359] Metadata directory not provided
I20260928 22:51:56.621336 32454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:51:56.621426 32454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:51:56.622466 32454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-1/data/instance:
uuid: "6fc5b62cd2534fd090ba8c61fdb3c421"
format_stamp: "Formatted at 2026-09-28 22:51:56 on dist-test-slave-56vw"
server_key: "d2fbd6564ca0fa3f5638e31cb3fef99a"
server_key_iv: "94f56c3cbea3dd56d164cc52f9928faf"
server_key_version: "encryptionkey@0"
I20260928 22:51:56.622778 32454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance:
uuid: "6fc5b62cd2534fd090ba8c61fdb3c421"
format_stamp: "Formatted at 2026-09-28 22:51:56 on dist-test-slave-56vw"
server_key: "d2fbd6564ca0fa3f5638e31cb3fef99a"
server_key_iv: "94f56c3cbea3dd56d164cc52f9928faf"
server_key_version: "encryptionkey@0"
I20260928 22:51:56.624182 32454 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:51:56.625027 32469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:56.625263 32454 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:51:56.625382 32454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-1/wal
uuid: "6fc5b62cd2534fd090ba8c61fdb3c421"
format_stamp: "Formatted at 2026-09-28 22:51:56 on dist-test-slave-56vw"
server_key: "d2fbd6564ca0fa3f5638e31cb3fef99a"
server_key_iv: "94f56c3cbea3dd56d164cc52f9928faf"
server_key_version: "encryptionkey@0"
I20260928 22:51:56.625468 32454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:51:56.633093 32454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:51:56.635520 32454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:51:56.635695 32454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:51:56.635938 32454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:51:56.636250 32454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:51:56.636301 32454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:56.636343 32454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:51:56.636371 32454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:56.642442 32454 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.130:33137
I20260928 22:51:56.642558 32582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.130:33137 every 8 connection(s)
I20260928 22:51:56.642891 32454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
I20260928 22:51:56.648173 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 32454
I20260928 22:51:56.648285 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance
I20260928 22:51:56.648499 16870 external_mini_cluster.cc:1428] Setting key f8d1fc7c668ad0157c12c93699d4d3b0
Sep 28 22:51:56 dist-test-slave-56vw krb5kdc[32213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635916, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.130@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:51:56.653393 32298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:56.643811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:32825 (local address 127.16.121.190:34005)
0928 22:51:56.643874 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:56.643876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:56.644044 (+ 168us) server_negotiation.cc:407] Connection header received
0928 22:51:56.644534 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:56.644536 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:56.644550 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:56.644619 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:51:56.645559 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:56.646084 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:56.647360 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:56.647568 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:56.647635 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:56.650458 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:51:56.650460 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:51:56.650461 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:51:56.650469 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:51:56.651979 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:56.652405 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:56.652406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:56.652407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:56.652468 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:56.652722 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:56.652723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:56.652724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:56.652854 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:51:56.652932 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:56.653207 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 22:51:56.653296 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 22:51:56.653604 32585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:56.643739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.130:32825)
0928 22:51:56.643912 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:56.643954 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 22:51:56.644449 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:51:56.644681 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:51:56.644685 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:51:56.645253 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:51:56.645465 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:56.645470 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:56.646296 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:56.646297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:56.647272 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:56.647275 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:56.647649 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:56.647650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:56.647789 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:56.648380 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:51:56.648389 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:51:56.650344 (+ 1955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:51:56.652078 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:56.652085 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:56.652087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:56.652319 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:56.652532 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:56.652532 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:56.652533 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:56.652645 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:56.652921 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:51:56.652928 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:51:56.653075 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 22:51:56.653188 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 22:51:56.653354 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
I20260928 22:51:56.654094 32583 heartbeater.cc:347] Connected to a master server at 127.16.121.190:34005
I20260928 22:51:56.654197 32583 heartbeater.cc:464] Registering TS with master...
I20260928 22:51:56.654388 32583 heartbeater.cc:511] Master 127.16.121.190:34005 requested a full tablet report, sending...
I20260928 22:51:56.654804 32258 ts_manager.cc:194] Registered new tserver with Master: 6fc5b62cd2534fd090ba8c61fdb3c421 (127.16.121.130:33137)
I20260928 22:51:56.655261 32258 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.130@KRBTEST.COM'} at 127.16.121.130:32825
WARNING: no policy specified for kudu/127.16.121.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab.
Entry for principal kudu/127.16.121.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab.
I20260928 22:51:56.678704 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.131:0
--local_ip_for_outbound_sockets=127.16.121.131
--webserver_interface=127.16.121.131
--webserver_port=0
--tserver_master_addrs=127.16.121.190:34005
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab
--principal=kudu/127.16.121.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:42335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:51:56.762960 32588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:51:56.763269 32588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:51:56.763298 32588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:51:56.764509 32588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:51:56.764551 32588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:51:56.764578 32588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.131
I20260928 22:51:56.766018 32588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:42335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab
--principal=kudu/127.16.121.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.121.131
--webserver_port=0
--tserver_master_addrs=127.16.121.190:34005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.131
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:51:56.766216 32588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:51:56.766413 32588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:51:56.771873 32596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:51:56.771880 32594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:51:56.771880 32593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:51:56.772320 32588 server_base.cc:1034] running on GCE node
I20260928 22:51:56.772560 32588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:51:56.772856 32588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:51:56.774024 32588 hybrid_clock.cc:648] HybridClock initialized: now 1790635916774012 us; error 52 us; skew 500 ppm
Sep 28 22:51:56 dist-test-slave-56vw krb5kdc[32213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635916, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:51:56.776757 32588 init.cc:377] Logged in from keytab as kudu/127.16.121.131@KRBTEST.COM (short username kudu)
I20260928 22:51:56.777204 32588 webserver.cc:460] Webserver started at http://127.16.121.131:35385/ using document root <none> and password file <none>
I20260928 22:51:56.777437 32588 fs_manager.cc:359] Metadata directory not provided
I20260928 22:51:56.777501 32588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:51:56.777652 32588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:51:56.778743 32588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-2/data/instance:
uuid: "61f7571e3b354ccd803db55846d546d5"
format_stamp: "Formatted at 2026-09-28 22:51:56 on dist-test-slave-56vw"
server_key: "327cf5621defd0a04990ef8b41443833"
server_key_iv: "09a34ed5d2210dfdfe704fa7da140c59"
server_key_version: "encryptionkey@0"
I20260928 22:51:56.779175 32588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance:
uuid: "61f7571e3b354ccd803db55846d546d5"
format_stamp: "Formatted at 2026-09-28 22:51:56 on dist-test-slave-56vw"
server_key: "327cf5621defd0a04990ef8b41443833"
server_key_iv: "09a34ed5d2210dfdfe704fa7da140c59"
server_key_version: "encryptionkey@0"
I20260928 22:51:56.780561 32588 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:51:56.781464 32603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:56.781652 32588 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:51:56.781766 32588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-2/wal
uuid: "61f7571e3b354ccd803db55846d546d5"
format_stamp: "Formatted at 2026-09-28 22:51:56 on dist-test-slave-56vw"
server_key: "327cf5621defd0a04990ef8b41443833"
server_key_iv: "09a34ed5d2210dfdfe704fa7da140c59"
server_key_version: "encryptionkey@0"
I20260928 22:51:56.781860 32588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:51:56.806371 32588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:51:56.808928 32588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:51:56.809053 32588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:51:56.809289 32588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:51:56.809573 32588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:51:56.809604 32588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:56.809628 32588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:51:56.809639 32588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:51:56.815850 32588 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.131:37823
I20260928 22:51:56.815923 32716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.131:37823 every 8 connection(s)
I20260928 22:51:56.816265 32588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
Sep 28 22:51:56 dist-test-slave-56vw krb5kdc[32213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635916, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.131@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:51:56.824112 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 32588
I20260928 22:51:56.824220 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance
I20260928 22:51:56.824404 16870 external_mini_cluster.cc:1428] Setting key 1856df4837c5fa8a63bac5a16b6e1219
I20260928 22:51:56.826354 32298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:56.817042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:41841 (local address 127.16.121.190:34005)
0928 22:51:56.817130 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:56.817132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:56.817394 (+ 262us) server_negotiation.cc:407] Connection header received
0928 22:51:56.817855 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:56.817856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:56.817867 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:56.817931 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:51:56.818843 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:56.819429 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:56.820438 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:56.820638 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:56.820705 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:56.822865 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:51:56.822866 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:51:56.822867 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:51:56.822874 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:51:56.824757 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:56.825269 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:56.825270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:56.825271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:56.825338 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:56.825597 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:56.825598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:56.825598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:56.825749 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:51:56.825817 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:56.826148 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 22:51:56.826287 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 22:51:56.826449 32719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:56.817024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.131:41841)
0928 22:51:56.817210 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:56.817301 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0928 22:51:56.817777 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:51:56.817984 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:51:56.817987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:51:56.818555 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:51:56.818748 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:56.818753 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:56.819515 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:56.819516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:56.820339 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:56.820342 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:56.820719 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:56.820720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:56.820824 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:56.821410 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:51:56.821418 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:51:56.822776 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:51:56.824842 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:56.824849 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:56.824852 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:56.825174 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:56.825407 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:51:56.825408 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:51:56.825409 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:51:56.825516 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:51:56.825834 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:51:56.825841 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:51:56.825971 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 22:51:56.826071 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 22:51:56.826216 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":105,"threads_started":1}
I20260928 22:51:56.827028 32717 heartbeater.cc:347] Connected to a master server at 127.16.121.190:34005
I20260928 22:51:56.827234 32717 heartbeater.cc:464] Registering TS with master...
I20260928 22:51:56.827430 32717 heartbeater.cc:511] Master 127.16.121.190:34005 requested a full tablet report, sending...
I20260928 22:51:56.827901 32258 ts_manager.cc:194] Registered new tserver with Master: 61f7571e3b354ccd803db55846d546d5 (127.16.121.131:37823)
I20260928 22:51:56.828298 32258 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.131@KRBTEST.COM'} at 127.16.121.131:41841
I20260928 22:51:56.836444 16870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:51:56.844193 32298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:56.837906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52908 (local address 127.16.121.190:34005)
0928 22:51:56.837975 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:56.837976 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:56.838031 (+ 55us) server_negotiation.cc:407] Connection header received
0928 22:51:56.838079 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:56.838080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:56.838088 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:56.838147 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:51:56.838747 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:56.839368 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:56.840138 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:56.840335 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:56.840395 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:56.841675 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:51:56.841676 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:51:56.841677 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:51:56.841683 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:51:56.843023 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:56.843375 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:56.843376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:56.843377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:56.843434 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:51:56.843603 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:51:56.843604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:51:56.843611 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:51:56.843764 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:51:56.843843 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:56.844032 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 22:51:56.844110 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 22:51:56.846168 32258 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52908:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:51:56.848121 32258 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:51:56.855233 32732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:56.851021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48582 (local address 127.16.121.131:37823)
0928 22:51:56.851260 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:56.851266 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:56.851278 (+ 12us) server_negotiation.cc:407] Connection header received
0928 22:51:56.851320 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:56.851323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:56.851341 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:56.851425 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:51:56.851690 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:56.852850 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:56.854214 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:56.854731 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:56.854822 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:56.854886 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:56.855096 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 22:51:56.855140 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":129,"threads_started":1}
I20260928 22:51:56.855285 32728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:56.850700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.130:33137 (local address 127.0.0.1:41248)
0928 22:51:56.850841 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:56.850888 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 22:51:56.850959 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:51:56.851551 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:51:56.851552 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:51:56.851555 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:51:56.851619 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:56.851621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:56.852962 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:56.852963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:56.854064 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:56.854067 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:56.854960 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:56.854961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:56.855026 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:56.855030 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:51:56.855091 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 22:51:56.855148 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":55,"thread_start_us":87,"threads_started":1}
I20260928 22:51:56.855298 32729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:56.850933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:37381 (local address 127.0.0.1:39754)
0928 22:51:56.851184 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:56.851194 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:51:56.851238 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:51:56.851671 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:51:56.851672 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:51:56.851674 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:51:56.851708 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:56.851709 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:56.852630 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:56.852631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:56.853967 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:56.853969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:56.855007 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:56.855008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:56.855090 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:56.855094 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:51:56.855139 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 22:51:56.855176 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":26,"thread_start_us":71,"threads_started":1}
I20260928 22:51:56.855504 32733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:56.851286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39754 (local address 127.16.121.129:37381)
0928 22:51:56.851501 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:56.851505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:56.851514 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:51:56.851550 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:56.851552 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:56.851566 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:56.851781 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:51:56.851840 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:56.852504 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:56.854065 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:56.854834 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:56.854917 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:56.854979 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:56.855341 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 22:51:56.855398 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":95,"threads_started":1}
I20260928 22:51:56.855281 32731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:56.850783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.131:37823 (local address 127.0.0.1:48582)
0928 22:51:56.851082 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:56.851094 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:51:56.851217 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:51:56.851529 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:51:56.851532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:51:56.851535 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:51:56.851622 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:56.851625 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:56.853092 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:56.853093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:56.854125 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:56.854126 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:56.854831 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:56.854832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:56.854932 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:56.854948 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 22:51:56.855034 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 22:51:56.855128 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":102,"threads_started":1}
I20260928 22:51:56.856135 32730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:56.850830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41248 (local address 127.16.121.130:33137)
0928 22:51:56.851380 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:56.851385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:56.851395 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:51:56.851435 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:56.851438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:56.851454 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:56.851710 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:51:56.851791 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:56.852850 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:56.854175 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:56.854867 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:56.855939 (+ 1072us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:56.856011 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:56.856033 (+ 22us) server_negotiation.cc:299] Negotiation successful
0928 22:51:56.856073 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":77,"threads_started":1}
I20260928 22:51:56.856240 32383 tablet_service.cc:1467] Processing CreateTablet for tablet 4f3b789471ef4358a57642807e02f893 (DEFAULT_TABLE table=test-table [id=99d314b94ce946219f51aa9cf184a12b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:51:56.856601 32383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f3b789471ef4358a57642807e02f893. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:51:56.856779 32517 tablet_service.cc:1467] Processing CreateTablet for tablet 4f3b789471ef4358a57642807e02f893 (DEFAULT_TABLE table=test-table [id=99d314b94ce946219f51aa9cf184a12b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:51:56.857046 32517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f3b789471ef4358a57642807e02f893. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:51:56.859390 32651 tablet_service.cc:1467] Processing CreateTablet for tablet 4f3b789471ef4358a57642807e02f893 (DEFAULT_TABLE table=test-table [id=99d314b94ce946219f51aa9cf184a12b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:51:56.859630 32734 tablet_bootstrap.cc:492] T 4f3b789471ef4358a57642807e02f893 P 64000c9c5f484909b35f01e352a21041: Bootstrap starting.
I20260928 22:51:56.859777 32651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f3b789471ef4358a57642807e02f893. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:51:56.859975 32735 tablet_bootstrap.cc:492] T 4f3b789471ef4358a57642807e02f893 P 6fc5b62cd2534fd090ba8c61fdb3c421: Bootstrap starting.
I20260928 22:51:56.860303 32734 tablet_bootstrap.cc:654] T 4f3b789471ef4358a57642807e02f893 P 64000c9c5f484909b35f01e352a21041: Neither blocks nor log segments found. Creating new log.
I20260928 22:51:56.860695 32735 tablet_bootstrap.cc:654] T 4f3b789471ef4358a57642807e02f893 P 6fc5b62cd2534fd090ba8c61fdb3c421: Neither blocks nor log segments found. Creating new log.
I20260928 22:51:56.860880 32734 log.cc:824] T 4f3b789471ef4358a57642807e02f893 P 64000c9c5f484909b35f01e352a21041: Log is configured to *not* fsync() on all Append() calls
I20260928 22:51:56.861260 32735 log.cc:824] T 4f3b789471ef4358a57642807e02f893 P 6fc5b62cd2534fd090ba8c61fdb3c421: Log is configured to *not* fsync() on all Append() calls
I20260928 22:51:56.861687 32734 tablet_bootstrap.cc:492] T 4f3b789471ef4358a57642807e02f893 P 64000c9c5f484909b35f01e352a21041: No bootstrap required, opened a new log
I20260928 22:51:56.861768 32734 ts_tablet_manager.cc:1397] T 4f3b789471ef4358a57642807e02f893 P 64000c9c5f484909b35f01e352a21041: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:51:56.862169 32735 tablet_bootstrap.cc:492] T 4f3b789471ef4358a57642807e02f893 P 6fc5b62cd2534fd090ba8c61fdb3c421: No bootstrap required, opened a new log
I20260928 22:51:56.862239 32735 ts_tablet_manager.cc:1397] T 4f3b789471ef4358a57642807e02f893 P 6fc5b62cd2534fd090ba8c61fdb3c421: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:51:56.863394 32738 tablet_bootstrap.cc:492] T 4f3b789471ef4358a57642807e02f893 P 61f7571e3b354ccd803db55846d546d5: Bootstrap starting.
I20260928 22:51:56.864094 32738 tablet_bootstrap.cc:654] T 4f3b789471ef4358a57642807e02f893 P 61f7571e3b354ccd803db55846d546d5: Neither blocks nor log segments found. Creating new log.
I20260928 22:51:56.864125 32735 raft_consensus.cc:348] T 4f3b789471ef4358a57642807e02f893 P 6fc5b62cd2534fd090ba8c61fdb3c421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc5b62cd2534fd090ba8c61fdb3c421" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 33137 } } peers { permanent_uuid: "61f7571e3b354ccd803db55846d546d5" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 37823 } } peers { permanent_uuid: "64000c9c5f484909b35f01e352a21041" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37381 } }
I20260928 22:51:56.864166 32734 raft_consensus.cc:348] T 4f3b789471ef4358a57642807e02f893 P 64000c9c5f484909b35f01e352a21041 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc5b62cd2534fd090ba8c61fdb3c421" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 33137 } } peers { permanent_uuid: "61f7571e3b354ccd803db55846d546d5" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 37823 } } peers { permanent_uuid: "64000c9c5f484909b35f01e352a21041" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37381 } }
I20260928 22:51:56.864351 32734 raft_consensus.cc:374] T 4f3b789471ef4358a57642807e02f893 P 64000c9c5f484909b35f01e352a21041 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:51:56.864351 32735 raft_consensus.cc:374] T 4f3b789471ef4358a57642807e02f893 P 6fc5b62cd2534fd090ba8c61fdb3c421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:51:56.864398 32735 raft_consensus.cc:729] T 4f3b789471ef4358a57642807e02f893 P 6fc5b62cd2534fd090ba8c61fdb3c421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fc5b62cd2534fd090ba8c61fdb3c421, State: Initialized, Role: FOLLOWER
I20260928 22:51:56.864399 32734 raft_consensus.cc:729] T 4f3b789471ef4358a57642807e02f893 P 64000c9c5f484909b35f01e352a21041 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64000c9c5f484909b35f01e352a21041, State: Initialized, Role: FOLLOWER
I20260928 22:51:56.864550 32735 consensus_queue.cc:260] T 4f3b789471ef4358a57642807e02f893 P 6fc5b62cd2534fd090ba8c61fdb3c421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc5b62cd2534fd090ba8c61fdb3c421" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 33137 } } peers { permanent_uuid: "61f7571e3b354ccd803db55846d546d5" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 37823 } } peers { permanent_uuid: "64000c9c5f484909b35f01e352a21041" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37381 } }
I20260928 22:51:56.864552 32734 consensus_queue.cc:260] T 4f3b789471ef4358a57642807e02f893 P 64000c9c5f484909b35f01e352a21041 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc5b62cd2534fd090ba8c61fdb3c421" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 33137 } } peers { permanent_uuid: "61f7571e3b354ccd803db55846d546d5" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 37823 } } peers { permanent_uuid: "64000c9c5f484909b35f01e352a21041" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37381 } }
I20260928 22:51:56.864663 32738 log.cc:824] T 4f3b789471ef4358a57642807e02f893 P 61f7571e3b354ccd803db55846d546d5: Log is configured to *not* fsync() on all Append() calls
I20260928 22:51:56.864841 32735 ts_tablet_manager.cc:1428] T 4f3b789471ef4358a57642807e02f893 P 6fc5b62cd2534fd090ba8c61fdb3c421: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 22:51:56.864861 32734 ts_tablet_manager.cc:1428] T 4f3b789471ef4358a57642807e02f893 P 64000c9c5f484909b35f01e352a21041: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:51:56.864991 32449 heartbeater.cc:503] Master 127.16.121.190:34005 was elected leader, sending a full tablet report...
I20260928 22:51:56.864998 32583 heartbeater.cc:503] Master 127.16.121.190:34005 was elected leader, sending a full tablet report...
I20260928 22:51:56.865546 32738 tablet_bootstrap.cc:492] T 4f3b789471ef4358a57642807e02f893 P 61f7571e3b354ccd803db55846d546d5: No bootstrap required, opened a new log
I20260928 22:51:56.865618 32738 ts_tablet_manager.cc:1397] T 4f3b789471ef4358a57642807e02f893 P 61f7571e3b354ccd803db55846d546d5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:51:56.867511 32738 raft_consensus.cc:348] T 4f3b789471ef4358a57642807e02f893 P 61f7571e3b354ccd803db55846d546d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc5b62cd2534fd090ba8c61fdb3c421" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 33137 } } peers { permanent_uuid: "61f7571e3b354ccd803db55846d546d5" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 37823 } } peers { permanent_uuid: "64000c9c5f484909b35f01e352a21041" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37381 } }
I20260928 22:51:56.867663 32738 raft_consensus.cc:374] T 4f3b789471ef4358a57642807e02f893 P 61f7571e3b354ccd803db55846d546d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:51:56.867686 32738 raft_consensus.cc:729] T 4f3b789471ef4358a57642807e02f893 P 61f7571e3b354ccd803db55846d546d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61f7571e3b354ccd803db55846d546d5, State: Initialized, Role: FOLLOWER
I20260928 22:51:56.867853 32738 consensus_queue.cc:260] T 4f3b789471ef4358a57642807e02f893 P 61f7571e3b354ccd803db55846d546d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc5b62cd2534fd090ba8c61fdb3c421" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 33137 } } peers { permanent_uuid: "61f7571e3b354ccd803db55846d546d5" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 37823 } } peers { permanent_uuid: "64000c9c5f484909b35f01e352a21041" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37381 } }
I20260928 22:51:56.868156 32738 ts_tablet_manager.cc:1428] T 4f3b789471ef4358a57642807e02f893 P 61f7571e3b354ccd803db55846d546d5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 22:51:56.868208 32717 heartbeater.cc:503] Master 127.16.121.190:34005 was elected leader, sending a full tablet report...
W20260928 22:51:56.893582 32584 tablet.cc:2369] T 4f3b789471ef4358a57642807e02f893 P 6fc5b62cd2534fd090ba8c61fdb3c421: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:51:56.927445 32742 raft_consensus.cc:482] T 4f3b789471ef4358a57642807e02f893 P 61f7571e3b354ccd803db55846d546d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:51:56.927577 32742 raft_consensus.cc:504] T 4f3b789471ef4358a57642807e02f893 P 61f7571e3b354ccd803db55846d546d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc5b62cd2534fd090ba8c61fdb3c421" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 33137 } } peers { permanent_uuid: "61f7571e3b354ccd803db55846d546d5" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 37823 } } peers { permanent_uuid: "64000c9c5f484909b35f01e352a21041" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37381 } }
I20260928 22:51:56.927994 32742 leader_election.cc:288] T 4f3b789471ef4358a57642807e02f893 P 61f7571e3b354ccd803db55846d546d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fc5b62cd2534fd090ba8c61fdb3c421 (127.16.121.130:33137), 64000c9c5f484909b35f01e352a21041 (127.16.121.129:37381)
I20260928 22:51:56.931676 32730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:56.928452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:49481 (local address 127.16.121.130:33137)
0928 22:51:56.928509 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:56.928510 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:56.928519 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:51:56.928549 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:56.928550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:56.928560 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:56.928620 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:51:56.928840 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:56.929635 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:56.930835 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:56.931312 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:56.931396 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:56.931406 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:56.931563 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 22:51:56.931600 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 22:51:56.931691 32733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:56.928464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:34369 (local address 127.16.121.129:37381)
0928 22:51:56.928519 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:56.928522 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:56.928641 (+ 119us) server_negotiation.cc:407] Connection header received
0928 22:51:56.928747 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:56.928748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:56.928759 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:56.928826 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:51:56.928990 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:56.929734 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:56.930855 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:56.931325 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:56.931408 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:56.931416 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:56.931576 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 22:51:56.931615 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 22:51:56.931720 32719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:56.928228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.130:33137 (local address 127.16.121.131:49481)
0928 22:51:56.928340 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:56.928353 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:51:56.928416 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:51:56.928640 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:51:56.928642 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:51:56.928644 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:51:56.928736 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:56.928739 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:56.929733 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:56.929735 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:56.930731 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:56.930734 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:56.931393 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:56.931395 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:56.931482 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:56.931485 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:51:56.931545 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 22:51:56.931590 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":2}
I20260928 22:51:56.931958 32743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:56.928392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:37381 (local address 127.16.121.131:34369)
0928 22:51:56.928579 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:56.928589 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:51:56.928666 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:51:56.928839 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:51:56.928840 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:51:56.928842 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:51:56.928911 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:56.928913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:56.929890 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:56.929891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:56.930742 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:51:56.930745 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:51:56.931406 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:51:56.931408 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:51:56.931480 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:56.931484 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:51:56.931569 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 22:51:56.931613 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":3,"thread_start_us":130,"threads_started":1}
I20260928 22:51:56.932102 32537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f3b789471ef4358a57642807e02f893" candidate_uuid: "61f7571e3b354ccd803db55846d546d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fc5b62cd2534fd090ba8c61fdb3c421" is_pre_election: true
I20260928 22:51:56.932278 32537 raft_consensus.cc:2454] T 4f3b789471ef4358a57642807e02f893 P 6fc5b62cd2534fd090ba8c61fdb3c421 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61f7571e3b354ccd803db55846d546d5 in term 0.
I20260928 22:51:56.932461 32403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f3b789471ef4358a57642807e02f893" candidate_uuid: "61f7571e3b354ccd803db55846d546d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64000c9c5f484909b35f01e352a21041" is_pre_election: true
I20260928 22:51:56.932551 32604 leader_election.cc:302] T 4f3b789471ef4358a57642807e02f893 P 61f7571e3b354ccd803db55846d546d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61f7571e3b354ccd803db55846d546d5, 6fc5b62cd2534fd090ba8c61fdb3c421; no voters:
I20260928 22:51:56.932622 32403 raft_consensus.cc:2454] T 4f3b789471ef4358a57642807e02f893 P 64000c9c5f484909b35f01e352a21041 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61f7571e3b354ccd803db55846d546d5 in term 0.
I20260928 22:51:56.932705 32742 raft_consensus.cc:2781] T 4f3b789471ef4358a57642807e02f893 P 61f7571e3b354ccd803db55846d546d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:51:56.932761 32742 raft_consensus.cc:482] T 4f3b789471ef4358a57642807e02f893 P 61f7571e3b354ccd803db55846d546d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:51:56.932789 32742 raft_consensus.cc:3037] T 4f3b789471ef4358a57642807e02f893 P 61f7571e3b354ccd803db55846d546d5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:51:56.933528 32742 raft_consensus.cc:504] T 4f3b789471ef4358a57642807e02f893 P 61f7571e3b354ccd803db55846d546d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc5b62cd2534fd090ba8c61fdb3c421" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 33137 } } peers { permanent_uuid: "61f7571e3b354ccd803db55846d546d5" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 37823 } } peers { permanent_uuid: "64000c9c5f484909b35f01e352a21041" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37381 } }
I20260928 22:51:56.933669 32742 leader_election.cc:288] T 4f3b789471ef4358a57642807e02f893 P 61f7571e3b354ccd803db55846d546d5 [CANDIDATE]: Term 1 election: Requested vote from peers 6fc5b62cd2534fd090ba8c61fdb3c421 (127.16.121.130:33137), 64000c9c5f484909b35f01e352a21041 (127.16.121.129:37381)
I20260928 22:51:56.933869 32403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f3b789471ef4358a57642807e02f893" candidate_uuid: "61f7571e3b354ccd803db55846d546d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64000c9c5f484909b35f01e352a21041"
I20260928 22:51:56.933876 32537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f3b789471ef4358a57642807e02f893" candidate_uuid: "61f7571e3b354ccd803db55846d546d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fc5b62cd2534fd090ba8c61fdb3c421"
I20260928 22:51:56.933964 32403 raft_consensus.cc:3037] T 4f3b789471ef4358a57642807e02f893 P 64000c9c5f484909b35f01e352a21041 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:51:56.933961 32537 raft_consensus.cc:3037] T 4f3b789471ef4358a57642807e02f893 P 6fc5b62cd2534fd090ba8c61fdb3c421 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:51:56.934744 32537 raft_consensus.cc:2454] T 4f3b789471ef4358a57642807e02f893 P 6fc5b62cd2534fd090ba8c61fdb3c421 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61f7571e3b354ccd803db55846d546d5 in term 1.
I20260928 22:51:56.934759 32403 raft_consensus.cc:2454] T 4f3b789471ef4358a57642807e02f893 P 64000c9c5f484909b35f01e352a21041 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61f7571e3b354ccd803db55846d546d5 in term 1.
I20260928 22:51:56.934904 32604 leader_election.cc:302] T 4f3b789471ef4358a57642807e02f893 P 61f7571e3b354ccd803db55846d546d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61f7571e3b354ccd803db55846d546d5, 6fc5b62cd2534fd090ba8c61fdb3c421; no voters:
I20260928 22:51:56.935012 32742 raft_consensus.cc:2781] T 4f3b789471ef4358a57642807e02f893 P 61f7571e3b354ccd803db55846d546d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:51:56.935210 32742 raft_consensus.cc:686] T 4f3b789471ef4358a57642807e02f893 P 61f7571e3b354ccd803db55846d546d5 [term 1 LEADER]: Becoming Leader. State: Replica: 61f7571e3b354ccd803db55846d546d5, State: Running, Role: LEADER
I20260928 22:51:56.935446 32742 consensus_queue.cc:237] T 4f3b789471ef4358a57642807e02f893 P 61f7571e3b354ccd803db55846d546d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc5b62cd2534fd090ba8c61fdb3c421" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 33137 } } peers { permanent_uuid: "61f7571e3b354ccd803db55846d546d5" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 37823 } } peers { permanent_uuid: "64000c9c5f484909b35f01e352a21041" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37381 } }
I20260928 22:51:56.936390 32257 catalog_manager.cc:5495] T 4f3b789471ef4358a57642807e02f893 P 61f7571e3b354ccd803db55846d546d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61f7571e3b354ccd803db55846d546d5 (127.16.121.131). New cstate: current_term: 1 leader_uuid: "61f7571e3b354ccd803db55846d546d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc5b62cd2534fd090ba8c61fdb3c421" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 33137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61f7571e3b354ccd803db55846d546d5" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 37823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64000c9c5f484909b35f01e352a21041" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37381 } health_report { overall_health: UNKNOWN } } }
I20260928 22:51:56.948087 32732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:56.945528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48596 (local address 127.16.121.131:37823)
0928 22:51:56.945584 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 22:51:56.945587 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:51:56.945596 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:51:56.945682 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:51:56.945683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:51:56.945704 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:51:56.945788 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:51:56.945950 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:56.946437 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:56.947275 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:51:56.947489 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:51:56.947572 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:51:56.947747 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:51:56.947859 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:51:56.947953 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 22:51:56.947972 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 22:51:56.947996 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 22:51:56.950166 32403 raft_consensus.cc:1261] T 4f3b789471ef4358a57642807e02f893 P 64000c9c5f484909b35f01e352a21041 [term 1 FOLLOWER]: Refusing update from remote peer 61f7571e3b354ccd803db55846d546d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:51:56.950166 32537 raft_consensus.cc:1261] T 4f3b789471ef4358a57642807e02f893 P 6fc5b62cd2534fd090ba8c61fdb3c421 [term 1 FOLLOWER]: Refusing update from remote peer 61f7571e3b354ccd803db55846d546d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:51:56.950583 32744 consensus_queue.cc:1035] T 4f3b789471ef4358a57642807e02f893 P 61f7571e3b354ccd803db55846d546d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fc5b62cd2534fd090ba8c61fdb3c421" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 33137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:51:56.950760 32742 consensus_queue.cc:1035] T 4f3b789471ef4358a57642807e02f893 P 61f7571e3b354ccd803db55846d546d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64000c9c5f484909b35f01e352a21041" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:51:56.953526 32749 mvcc.cc:205] Tried to move back new op lower bound from 7334444715824750592 to 7334444715768401920. Current Snapshot: MvccSnapshot[applied={T|T < 7334444715824750592}]
I20260928 22:51:56.956483 32258 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:51:56.956583 32258 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:51:56.957458 32258 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c: Sending DeleteTablet for 3 replicas of tablet 4f3b789471ef4358a57642807e02f893
I20260928 22:51:56.957819 32651 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f3b789471ef4358a57642807e02f893 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:51:56 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:48582
I20260928 22:51:56.957805 32517 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f3b789471ef4358a57642807e02f893 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:51:56 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:41248
I20260928 22:51:56.957901 32383 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f3b789471ef4358a57642807e02f893 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:51:56 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:39754
I20260928 22:51:56.958061 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 32320
I20260928 22:51:56.958132 32758 tablet_replica.cc:330] T 4f3b789471ef4358a57642807e02f893 P 64000c9c5f484909b35f01e352a21041: stopping tablet replica
I20260928 22:51:56.959206 32757 tablet_replica.cc:330] T 4f3b789471ef4358a57642807e02f893 P 6fc5b62cd2534fd090ba8c61fdb3c421: stopping tablet replica
I20260928 22:51:56.959355 32757 raft_consensus.cc:2229] T 4f3b789471ef4358a57642807e02f893 P 6fc5b62cd2534fd090ba8c61fdb3c421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:51:56.959477 32757 raft_consensus.cc:2258] T 4f3b789471ef4358a57642807e02f893 P 6fc5b62cd2534fd090ba8c61fdb3c421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:51:56.959966 32757 ts_tablet_manager.cc:1905] T 4f3b789471ef4358a57642807e02f893 P 6fc5b62cd2534fd090ba8c61fdb3c421: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:51:56.961580 32757 ts_tablet_manager.cc:1918] T 4f3b789471ef4358a57642807e02f893 P 6fc5b62cd2534fd090ba8c61fdb3c421: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:51:56.961690 32757 log.cc:1196] T 4f3b789471ef4358a57642807e02f893 P 6fc5b62cd2534fd090ba8c61fdb3c421: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-1/wal/wals/4f3b789471ef4358a57642807e02f893
I20260928 22:51:56.962003 32759 tablet_replica.cc:330] T 4f3b789471ef4358a57642807e02f893 P 61f7571e3b354ccd803db55846d546d5: stopping tablet replica
I20260928 22:51:56.962065 32757 ts_tablet_manager.cc:1939] T 4f3b789471ef4358a57642807e02f893 P 6fc5b62cd2534fd090ba8c61fdb3c421: Deleting consensus metadata
I20260928 22:51:56.962216 32759 raft_consensus.cc:2229] T 4f3b789471ef4358a57642807e02f893 P 61f7571e3b354ccd803db55846d546d5 [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:51:56.962491 32759 raft_consensus.cc:2258] T 4f3b789471ef4358a57642807e02f893 P 61f7571e3b354ccd803db55846d546d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:51:56.962536 32242 catalog_manager.cc:4841] TS 6fc5b62cd2534fd090ba8c61fdb3c421 (127.16.121.130:33137): tablet 4f3b789471ef4358a57642807e02f893 (table test-table [id=99d314b94ce946219f51aa9cf184a12b]) successfully deleted
W20260928 22:51:56.962980 32242 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.121.129:37381 (error 108)
I20260928 22:51:56.963102 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 32454
I20260928 22:51:56.963331 32759 ts_tablet_manager.cc:1905] T 4f3b789471ef4358a57642807e02f893 P 61f7571e3b354ccd803db55846d546d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:51:56.965011 32759 ts_tablet_manager.cc:1918] T 4f3b789471ef4358a57642807e02f893 P 61f7571e3b354ccd803db55846d546d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:51:56.965096 32759 log.cc:1196] T 4f3b789471ef4358a57642807e02f893 P 61f7571e3b354ccd803db55846d546d5: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-2/wal/wals/4f3b789471ef4358a57642807e02f893
I20260928 22:51:56.965459 32759 ts_tablet_manager.cc:1939] T 4f3b789471ef4358a57642807e02f893 P 61f7571e3b354ccd803db55846d546d5: Deleting consensus metadata
I20260928 22:51:56.965894 32244 catalog_manager.cc:4841] TS 61f7571e3b354ccd803db55846d546d5 (127.16.121.131:37823): tablet 4f3b789471ef4358a57642807e02f893 (table test-table [id=99d314b94ce946219f51aa9cf184a12b]) successfully deleted
I20260928 22:51:56.970793 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 32588
I20260928 22:51:56.972688 32731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:51:56.972527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:37381 (local address 127.0.0.1:39758)
0928 22:51:56.972591 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 22:51:56.972652 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.129:37381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 22:51:56.972851 32242 catalog_manager.cc:4584] TS 64000c9c5f484909b35f01e352a21041 (127.16.121.129:37381): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4f3b789471ef4358a57642807e02f893: Network error: Client connection negotiation failed: client connection to 127.16.121.129:37381: connect: Connection refused (error 111)
I20260928 22:51:56.976166 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 32226
I20260928 22:52:16.982227 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:34005
--webserver_interface=127.16.121.190
--webserver_port=37533
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:42335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:34005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:52:17.065114 32761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:52:17.065380 32761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:52:17.065419 32761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:52:17.066609 32761 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 22:52:17.066676 32761 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 22:52:17.066699 32761 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 22:52:17.066715 32761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:52:17.066744 32761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:52:17.066776 32761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:52:17.066799 32761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:52:17.066826 32761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:52:17.068387 32761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:42335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:34005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:34005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.121.190
--webserver_port=37533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:52:17.068681 32761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:52:17.068910 32761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:52:17.078274 32767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:52:17.078280 300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:52:17.078280 302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:52:17.078320 32761 server_base.cc:1034] running on GCE node
I20260928 22:52:17.078785 32761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:52:17.079170 32761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:52:17.080334 32761 hybrid_clock.cc:648] HybridClock initialized: now 1790635937080321 us; error 29 us; skew 500 ppm
Sep 28 22:52:17 dist-test-slave-56vw krb5kdc[32213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635937, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:52:17.083151 32761 init.cc:377] Logged in from keytab as kudu/127.16.121.190@KRBTEST.COM (short username kudu)
I20260928 22:52:17.083724 32761 webserver.cc:460] Webserver started at http://127.16.121.190:37533/ using document root <none> and password file <none>
I20260928 22:52:17.083981 32761 fs_manager.cc:359] Metadata directory not provided
I20260928 22:52:17.084071 32761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:52:17.085547 32761 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:52:17.086333 309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:17.086527 32761 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:52:17.086638 32761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "6c9fdbbae88b4180bcbb87227139de8c"
format_stamp: "Formatted at 2026-09-28 22:51:56 on dist-test-slave-56vw"
server_key: "6f6ce0f70f5a723c4b3876801326714f"
server_key_iv: "db908605c2ebfc5e870579b9e8dd415f"
server_key_version: "encryptionkey@0"
I20260928 22:52:17.087129 32761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:52:17.105005 32761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:52:17.107615 32761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:52:17.107740 32761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:52:17.111896 32761 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.190:34005
I20260928 22:52:17.111935 361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.190:34005 every 8 connection(s)
I20260928 22:52:17.112349 32761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:52:17.113906 362 sys_catalog.cc:263] Verifying existing consensus state
I20260928 22:52:17.114634 362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c: Bootstrap starting.
I20260928 22:52:17.117722 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 32761
I20260928 22:52:17.118065 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.129:37381
--local_ip_for_outbound_sockets=127.16.121.129
--tserver_master_addrs=127.16.121.190:34005
--webserver_port=33913
--webserver_interface=127.16.121.129
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab
--principal=kudu/127.16.121.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:42335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:52:17.124580 362 log.cc:824] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c: Log is configured to *not* fsync() on all Append() calls
I20260928 22:52:17.127606 362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 22:52:17.127995 362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c: Bootstrap complete.
I20260928 22:52:17.131587 362 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c9fdbbae88b4180bcbb87227139de8c" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 34005 } }
I20260928 22:52:17.131924 362 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c9fdbbae88b4180bcbb87227139de8c, State: Initialized, Role: FOLLOWER
I20260928 22:52:17.132186 362 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c9fdbbae88b4180bcbb87227139de8c" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 34005 } }
I20260928 22:52:17.132262 362 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:52:17.132306 362 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:52:17.132354 362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [term 1 FOLLOWER]: Advancing to term 2
I20260928 22:52:17.133746 362 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c9fdbbae88b4180bcbb87227139de8c" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 34005 } }
I20260928 22:52:17.133864 362 leader_election.cc:302] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c9fdbbae88b4180bcbb87227139de8c; no voters:
I20260928 22:52:17.134047 362 leader_election.cc:288] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 22:52:17.134142 366 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [term 2 FOLLOWER]: Leader election won for term 2
I20260928 22:52:17.134305 362 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:52:17.134351 366 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [term 2 LEADER]: Becoming Leader. State: Replica: 6c9fdbbae88b4180bcbb87227139de8c, State: Running, Role: LEADER
I20260928 22:52:17.134505 366 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c9fdbbae88b4180bcbb87227139de8c" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 34005 } }
I20260928 22:52:17.134857 367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6c9fdbbae88b4180bcbb87227139de8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c9fdbbae88b4180bcbb87227139de8c" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 34005 } } }
I20260928 22:52:17.134940 367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [sys.catalog]: This master's current role is: LEADER
I20260928 22:52:17.135228 368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c9fdbbae88b4180bcbb87227139de8c. Latest consensus state: current_term: 2 leader_uuid: "6c9fdbbae88b4180bcbb87227139de8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c9fdbbae88b4180bcbb87227139de8c" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 34005 } } }
I20260928 22:52:17.135308 368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [sys.catalog]: This master's current role is: LEADER
I20260928 22:52:17.135318 375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:52:17.136181 375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:52:17.136288 375 catalog_manager.cc:1259] Loaded cluster ID: c471a56f6b4143c9b6e4cfa2a6484473
I20260928 22:52:17.136371 375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:52:17.137367 375 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:52:17.137569 375 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c: Loaded TSK: 0
I20260928 22:52:17.151399 375 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c: Generated new TSK 1
I20260928 22:52:17.151548 375 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 22:52:17.218894 364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:52:17.219206 364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:52:17.219254 364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:52:17.220521 364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:52:17.220592 364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:52:17.220639 364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.129
I20260928 22:52:17.222198 364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:42335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab
--principal=kudu/127.16.121.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.129:37381
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.121.129
--webserver_port=33913
--tserver_master_addrs=127.16.121.190:34005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.129
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:52:17.222473 364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:52:17.222716 364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:52:17.229151 389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:52:17.229153 390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:52:17.229410 364 server_base.cc:1034] running on GCE node
W20260928 22:52:17.229153 392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:52:17.229843 364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:52:17.230161 364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:52:17.231359 364 hybrid_clock.cc:648] HybridClock initialized: now 1790635937231343 us; error 33 us; skew 500 ppm
Sep 28 22:52:17 dist-test-slave-56vw krb5kdc[32213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635937, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:52:17.234313 364 init.cc:377] Logged in from keytab as kudu/127.16.121.129@KRBTEST.COM (short username kudu)
I20260928 22:52:17.235143 364 webserver.cc:460] Webserver started at http://127.16.121.129:33913/ using document root <none> and password file <none>
I20260928 22:52:17.235386 364 fs_manager.cc:359] Metadata directory not provided
I20260928 22:52:17.235482 364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:52:17.237126 364 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:52:17.238003 399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:17.238202 364 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:52:17.238310 364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-0/wal
uuid: "64000c9c5f484909b35f01e352a21041"
format_stamp: "Formatted at 2026-09-28 22:51:56 on dist-test-slave-56vw"
server_key: "3979c4e8dcb7661137b395086cd04247"
server_key_iv: "9010955251a31455e4f4fbb931c618df"
server_key_version: "encryptionkey@0"
I20260928 22:52:17.238760 364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:52:17.264408 364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:52:17.266767 364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:52:17.266959 364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:52:17.267249 364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:52:17.267841 406 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 22:52:17.269323 364 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 22:52:17.269380 364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 22:52:17.269408 364 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 22:52:17.270249 364 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 22:52:17.270314 364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 22:52:17.270342 406 tablet_bootstrap.cc:492] T 4f3b789471ef4358a57642807e02f893 P 64000c9c5f484909b35f01e352a21041: Bootstrap starting.
I20260928 22:52:17.279246 364 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.129:37381
I20260928 22:52:17.279876 364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
I20260928 22:52:17.282752 513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.129:37381 every 8 connection(s)
I20260928 22:52:17.283144 406 log.cc:824] T 4f3b789471ef4358a57642807e02f893 P 64000c9c5f484909b35f01e352a21041: Log is configured to *not* fsync() on all Append() calls
I20260928 22:52:17.284672 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 364
I20260928 22:52:17.285027 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.130:33137
--local_ip_for_outbound_sockets=127.16.121.130
--tserver_master_addrs=127.16.121.190:34005
--webserver_port=44219
--webserver_interface=127.16.121.130
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab
--principal=kudu/127.16.121.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:42335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:52:17.285894 406 tablet_bootstrap.cc:492] T 4f3b789471ef4358a57642807e02f893 P 64000c9c5f484909b35f01e352a21041: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 22:52:17.286224 406 tablet_bootstrap.cc:492] T 4f3b789471ef4358a57642807e02f893 P 64000c9c5f484909b35f01e352a21041: Bootstrap complete.
I20260928 22:52:17.287498 406 ts_tablet_manager.cc:1397] T 4f3b789471ef4358a57642807e02f893 P 64000c9c5f484909b35f01e352a21041: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.011s
I20260928 22:52:17.288678 406 raft_consensus.cc:348] T 4f3b789471ef4358a57642807e02f893 P 64000c9c5f484909b35f01e352a21041 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc5b62cd2534fd090ba8c61fdb3c421" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 33137 } } peers { permanent_uuid: "61f7571e3b354ccd803db55846d546d5" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 37823 } } peers { permanent_uuid: "64000c9c5f484909b35f01e352a21041" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37381 } }
I20260928 22:52:17.289021 406 raft_consensus.cc:729] T 4f3b789471ef4358a57642807e02f893 P 64000c9c5f484909b35f01e352a21041 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64000c9c5f484909b35f01e352a21041, State: Initialized, Role: FOLLOWER
I20260928 22:52:17.289158 406 consensus_queue.cc:260] T 4f3b789471ef4358a57642807e02f893 P 64000c9c5f484909b35f01e352a21041 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc5b62cd2534fd090ba8c61fdb3c421" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 33137 } } peers { permanent_uuid: "61f7571e3b354ccd803db55846d546d5" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 37823 } } peers { permanent_uuid: "64000c9c5f484909b35f01e352a21041" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37381 } }
I20260928 22:52:17.289475 406 ts_tablet_manager.cc:1428] T 4f3b789471ef4358a57642807e02f893 P 64000c9c5f484909b35f01e352a21041: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
Sep 28 22:52:17 dist-test-slave-56vw krb5kdc[32213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635937, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:52:17.296203 517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:17.280782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:47637 (local address 127.16.121.190:34005)
0928 22:52:17.280990 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:17.281000 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:17.282626 (+ 1626us) server_negotiation.cc:407] Connection header received
0928 22:52:17.283499 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:17.283507 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:17.283581 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:17.283772 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:52:17.284623 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:17.285977 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:17.287838 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:17.288560 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:17.288710 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:17.291510 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:52:17.291514 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:52:17.291514 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:52:17.291527 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:52:17.293862 (+ 2335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:52:17.294598 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:52:17.294601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:52:17.294601 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:52:17.294710 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:52:17.295159 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:52:17.295160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:52:17.295161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:52:17.295418 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:52:17.295485 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:17.295928 (+ 443us) server_negotiation.cc:299] Negotiation successful
0928 22:52:17.296051 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":108,"threads_started":1}
I20260928 22:52:17.296458 516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:17.280644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.129:47637)
0928 22:52:17.282394 (+ 1750us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:17.282438 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 22:52:17.283229 (+ 791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:52:17.283802 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:52:17.283807 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:52:17.284171 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:52:17.284472 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:17.284479 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:17.286158 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:17.286159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:17.287733 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:17.287735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:17.288694 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:17.288696 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:17.288832 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:17.289564 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:52:17.289574 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:52:17.291380 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:52:17.293972 (+ 2592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:52:17.293984 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:52:17.293987 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:52:17.294424 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:52:17.294792 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:52:17.294793 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:52:17.294793 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:52:17.294978 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:52:17.295502 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:52:17.295510 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:52:17.295721 (+ 211us) client_negotiation.cc:769] Sending connection context
0928 22:52:17.295859 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 22:52:17.296092 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1704,"thread_start_us":92,"threads_started":1}
I20260928 22:52:17.297351 514 heartbeater.cc:347] Connected to a master server at 127.16.121.190:34005
I20260928 22:52:17.297458 514 heartbeater.cc:464] Registering TS with master...
I20260928 22:52:17.297683 514 heartbeater.cc:511] Master 127.16.121.190:34005 requested a full tablet report, sending...
I20260928 22:52:17.298449 326 ts_manager.cc:194] Registered new tserver with Master: 64000c9c5f484909b35f01e352a21041 (127.16.121.129:37381)
I20260928 22:52:17.299692 326 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.129@KRBTEST.COM'} at 127.16.121.129:47637
I20260928 22:52:17.300690 514 heartbeater.cc:503] Master 127.16.121.190:34005 was elected leader, sending a full tablet report...
Sep 28 22:52:17 dist-test-slave-56vw krb5kdc[32213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635937, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for kudu/127.16.121.129@KRBTEST.COM
I20260928 22:52:17.309345 521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:17.299100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:37381 (local address 127.0.0.1:44560)
0928 22:52:17.299291 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:17.299321 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 22:52:17.299385 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:52:17.299679 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:52:17.299688 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:52:17.300142 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:52:17.300231 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:17.300234 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:17.301853 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:17.301855 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:17.302851 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:17.302854 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:17.303409 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:17.303411 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:17.303494 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:17.304033 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:52:17.304038 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:52:17.305639 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:52:17.307932 (+ 2293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:52:17.307941 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:52:17.307942 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:52:17.308206 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:52:17.308493 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:52:17.308494 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:52:17.308494 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:52:17.308570 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:52:17.308988 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:52:17.308996 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:52:17.309054 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 22:52:17.309141 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 22:52:17.309259 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":71,"threads_started":1}
I20260928 22:52:17.309401 522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:17.299181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44560 (local address 127.16.121.129:37381)
0928 22:52:17.299516 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:17.299521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:17.299529 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:52:17.299560 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:17.299562 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:17.299575 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:17.299643 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:52:17.300410 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:17.301750 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:17.302940 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:17.303335 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:17.303399 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:17.305737 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:52:17.305748 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:52:17.305748 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:52:17.305756 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:52:17.307820 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:52:17.308328 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:52:17.308331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:52:17.308332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:52:17.308399 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:52:17.308653 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:52:17.308654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:52:17.308654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:52:17.308857 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:52:17.308946 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:17.309192 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 22:52:17.309299 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":85,"threads_started":1}
I20260928 22:52:17.309970 448 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f3b789471ef4358a57642807e02f893 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:51:56 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:44560
I20260928 22:52:17.310642 523 tablet_replica.cc:330] stopping tablet replica
I20260928 22:52:17.310773 523 raft_consensus.cc:2229] T 4f3b789471ef4358a57642807e02f893 P 64000c9c5f484909b35f01e352a21041 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:52:17.310858 523 raft_consensus.cc:2258] T 4f3b789471ef4358a57642807e02f893 P 64000c9c5f484909b35f01e352a21041 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:52:17.311249 523 ts_tablet_manager.cc:1905] T 4f3b789471ef4358a57642807e02f893 P 64000c9c5f484909b35f01e352a21041: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:52:17.313589 523 ts_tablet_manager.cc:1918] T 4f3b789471ef4358a57642807e02f893 P 64000c9c5f484909b35f01e352a21041: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:52:17.313725 523 log.cc:1196] T 4f3b789471ef4358a57642807e02f893 P 64000c9c5f484909b35f01e352a21041: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-0/wal/wals/4f3b789471ef4358a57642807e02f893
I20260928 22:52:17.313956 523 ts_tablet_manager.cc:1939] T 4f3b789471ef4358a57642807e02f893 P 64000c9c5f484909b35f01e352a21041: Deleting consensus metadata
I20260928 22:52:17.314554 310 catalog_manager.cc:4841] TS 64000c9c5f484909b35f01e352a21041 (127.16.121.129:37381): tablet 4f3b789471ef4358a57642807e02f893 (table test-table [id=99d314b94ce946219f51aa9cf184a12b]) successfully deleted
W20260928 22:52:17.379333 519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:52:17.379571 519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:52:17.379599 519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:52:17.380963 519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:52:17.381012 519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:52:17.381043 519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.130
I20260928 22:52:17.382668 519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:42335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab
--principal=kudu/127.16.121.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.130:33137
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.121.130
--webserver_port=44219
--tserver_master_addrs=127.16.121.190:34005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.130
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:52:17.382963 519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:52:17.383247 519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:52:17.389002 529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:52:17.389047 531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:52:17.389132 519 server_base.cc:1034] running on GCE node
W20260928 22:52:17.389027 528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:52:17.389691 519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:52:17.389984 519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:52:17.391119 519 hybrid_clock.cc:648] HybridClock initialized: now 1790635937391095 us; error 30 us; skew 500 ppm
Sep 28 22:52:17 dist-test-slave-56vw krb5kdc[32213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635937, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:52:17.393934 519 init.cc:377] Logged in from keytab as kudu/127.16.121.130@KRBTEST.COM (short username kudu)
I20260928 22:52:17.394400 519 webserver.cc:460] Webserver started at http://127.16.121.130:44219/ using document root <none> and password file <none>
I20260928 22:52:17.394650 519 fs_manager.cc:359] Metadata directory not provided
I20260928 22:52:17.394750 519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:52:17.396282 519 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:52:17.397006 538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:17.397203 519 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:52:17.397320 519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-1/wal
uuid: "6fc5b62cd2534fd090ba8c61fdb3c421"
format_stamp: "Formatted at 2026-09-28 22:51:56 on dist-test-slave-56vw"
server_key: "d2fbd6564ca0fa3f5638e31cb3fef99a"
server_key_iv: "94f56c3cbea3dd56d164cc52f9928faf"
server_key_version: "encryptionkey@0"
I20260928 22:52:17.397747 519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:52:17.412171 519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:52:17.414808 519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:52:17.414989 519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:52:17.415313 519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:52:17.415653 519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:52:17.415720 519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:17.415766 519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:52:17.415813 519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:17.422427 519 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.130:33137
I20260928 22:52:17.422469 651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.130:33137 every 8 connection(s)
I20260928 22:52:17.422853 519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
Sep 28 22:52:17 dist-test-slave-56vw krb5kdc[32213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635937, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.130@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:52:17.431648 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 519
I20260928 22:52:17.431967 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.131:37823
--local_ip_for_outbound_sockets=127.16.121.131
--tserver_master_addrs=127.16.121.190:34005
--webserver_port=35385
--webserver_interface=127.16.121.131
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab
--principal=kudu/127.16.121.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:42335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:52:17.432788 517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:17.423693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:47713 (local address 127.16.121.190:34005)
0928 22:52:17.423779 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:17.423781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:17.423964 (+ 183us) server_negotiation.cc:407] Connection header received
0928 22:52:17.424566 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:17.424567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:17.424580 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:17.424655 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:52:17.425654 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:17.426167 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:17.427350 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:17.427583 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:17.427672 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:17.430014 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:52:17.430015 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:52:17.430015 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:52:17.430022 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:52:17.431335 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:52:17.431732 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:52:17.431734 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:52:17.431735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:52:17.431802 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:52:17.432050 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:52:17.432051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:52:17.432052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:52:17.432206 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:52:17.432273 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:17.432541 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 22:52:17.432645 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 22:52:17.432946 654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:17.423645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.130:47713)
0928 22:52:17.423811 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:17.423850 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 22:52:17.424469 (+ 619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:52:17.424722 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:52:17.424725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:52:17.425339 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:52:17.425553 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:17.425557 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:17.426253 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:17.426254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:17.427270 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:17.427272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:17.427734 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:17.427735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:17.427841 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:17.428437 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:52:17.428445 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:52:17.429912 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:52:17.431405 (+ 1493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:52:17.431411 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:52:17.431413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:52:17.431638 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:52:17.431867 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:52:17.431867 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:52:17.431868 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:52:17.431973 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:52:17.432276 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:52:17.432281 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:52:17.432407 (+ 126us) client_negotiation.cc:769] Sending connection context
0928 22:52:17.432500 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 22:52:17.432627 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":102,"threads_started":1}
I20260928 22:52:17.433487 652 heartbeater.cc:347] Connected to a master server at 127.16.121.190:34005
I20260928 22:52:17.433606 652 heartbeater.cc:464] Registering TS with master...
I20260928 22:52:17.433820 652 heartbeater.cc:511] Master 127.16.121.190:34005 requested a full tablet report, sending...
I20260928 22:52:17.434273 326 ts_manager.cc:194] Registered new tserver with Master: 6fc5b62cd2534fd090ba8c61fdb3c421 (127.16.121.130:33137)
I20260928 22:52:17.434723 326 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.130@KRBTEST.COM'} at 127.16.121.130:47713
W20260928 22:52:17.514890 655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:52:17.515151 655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:52:17.515178 655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:52:17.516427 655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:52:17.516472 655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:52:17.516500 655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.131
I20260928 22:52:17.518076 655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:42335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab
--principal=kudu/127.16.121.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.131:37823
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.121.131
--webserver_port=35385
--tserver_master_addrs=127.16.121.190:34005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.131
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:52:17.518342 655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:52:17.518599 655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:52:17.523787 661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:52:17.523823 663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:52:17.523831 660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:52:17.524115 655 server_base.cc:1034] running on GCE node
I20260928 22:52:17.524296 655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:52:17.524492 655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:52:17.525665 655 hybrid_clock.cc:648] HybridClock initialized: now 1790635937525644 us; error 33 us; skew 500 ppm
Sep 28 22:52:17 dist-test-slave-56vw krb5kdc[32213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635937, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:52:17.528443 655 init.cc:377] Logged in from keytab as kudu/127.16.121.131@KRBTEST.COM (short username kudu)
I20260928 22:52:17.528973 655 webserver.cc:460] Webserver started at http://127.16.121.131:35385/ using document root <none> and password file <none>
I20260928 22:52:17.529250 655 fs_manager.cc:359] Metadata directory not provided
I20260928 22:52:17.529335 655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:52:17.530771 655 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:52:17.531525 670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:17.531697 655 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 22:52:17.531800 655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-2/wal
uuid: "61f7571e3b354ccd803db55846d546d5"
format_stamp: "Formatted at 2026-09-28 22:51:56 on dist-test-slave-56vw"
server_key: "327cf5621defd0a04990ef8b41443833"
server_key_iv: "09a34ed5d2210dfdfe704fa7da140c59"
server_key_version: "encryptionkey@0"
I20260928 22:52:17.532186 655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:52:17.546715 655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:52:17.548857 655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:52:17.549031 655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:52:17.549280 655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:52:17.549598 655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:52:17.549656 655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:17.549693 655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:52:17.549721 655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:17.556084 655 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.131:37823
I20260928 22:52:17.556137 783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.131:37823 every 8 connection(s)
I20260928 22:52:17.556504 655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
I20260928 22:52:17.557531 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 655
Sep 28 22:52:17 dist-test-slave-56vw krb5kdc[32213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635937, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.131@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:52:17.567395 787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:17.557977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55548 (local address 127.16.121.190:34005)
0928 22:52:17.558116 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:17.558118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:17.558215 (+ 97us) server_negotiation.cc:407] Connection header received
0928 22:52:17.558287 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:17.558287 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:17.558298 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:17.558363 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:52:17.559240 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:17.559811 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:17.560684 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:17.560920 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:17.561006 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:17.562679 (+ 1673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:52:17.562680 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:52:17.562681 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:52:17.562688 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:52:17.565956 (+ 3268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:52:17.566354 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:52:17.566355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:52:17.566355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:52:17.566406 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:52:17.566722 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:52:17.566723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:52:17.566723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:52:17.566901 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:52:17.567040 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:17.567184 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 22:52:17.567272 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":58,"threads_started":1}
I20260928 22:52:17.568048 517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:17.557347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:34837 (local address 127.16.121.190:34005)
0928 22:52:17.557435 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:17.557438 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:17.557715 (+ 277us) server_negotiation.cc:407] Connection header received
0928 22:52:17.558430 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:17.558430 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:17.558436 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:17.558465 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:52:17.559672 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:17.560145 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:17.561291 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:17.561759 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:17.561825 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:17.564614 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:52:17.564615 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:52:17.564616 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:52:17.564623 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:52:17.566181 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:52:17.566722 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:52:17.566723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:52:17.566723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:52:17.566822 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:52:17.567154 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:52:17.567155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:52:17.567155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:52:17.567277 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:52:17.567346 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:17.567874 (+ 528us) server_negotiation.cc:299] Negotiation successful
0928 22:52:17.567975 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 22:52:17.568326 786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:17.557332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.131:34837)
0928 22:52:17.557565 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:17.557621 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 22:52:17.558357 (+ 736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:52:17.558571 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:52:17.558576 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:52:17.559369 (+ 793us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:52:17.559568 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:17.559573 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:17.560248 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:17.560249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:17.561133 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:17.561136 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:17.561890 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:17.561891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:17.562094 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:17.562648 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:52:17.562660 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:52:17.564499 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:52:17.566260 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:52:17.566268 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:52:17.566270 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:52:17.566631 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:52:17.566900 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:52:17.566901 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:52:17.566901 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:52:17.567095 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:52:17.567360 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:52:17.567367 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:52:17.567703 (+ 336us) client_negotiation.cc:769] Sending connection context
0928 22:52:17.567820 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 22:52:17.568030 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":128,"threads_started":1}
I20260928 22:52:17.568769 784 heartbeater.cc:347] Connected to a master server at 127.16.121.190:34005
I20260928 22:52:17.568871 784 heartbeater.cc:464] Registering TS with master...
I20260928 22:52:17.569020 784 heartbeater.cc:511] Master 127.16.121.190:34005 requested a full tablet report, sending...
I20260928 22:52:17.569350 326 ts_manager.cc:194] Registered new tserver with Master: 61f7571e3b354ccd803db55846d546d5 (127.16.121.131:37823)
I20260928 22:52:17.569698 326 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.131@KRBTEST.COM'} at 127.16.121.131:34837
I20260928 22:52:17.579640 16870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:52:17.582521 517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:17.580063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55554 (local address 127.16.121.190:34005)
0928 22:52:17.580121 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:17.580123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:17.580235 (+ 112us) server_negotiation.cc:407] Connection header received
0928 22:52:17.580296 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:17.580296 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:17.580336 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:17.580400 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:52:17.580530 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:17.581057 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:17.581826 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:17.582017 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:17.582090 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:17.582279 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:52:17.582329 (+ 50us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 22:52:17.582465 (+ 136us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55554: token expired
Metrics: {"server-negotiator.queue_time_us":33}
W20260928 22:52:17.582607 517 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55554: token expired
W20260928 22:52:17.582770 32721 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.16.121.190:34005)
I20260928 22:52:17.589496 517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:17.583023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55568 (local address 127.16.121.190:34005)
0928 22:52:17.583091 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:17.583093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:17.583103 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:52:17.583163 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:17.583164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:17.583171 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:17.583218 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:52:17.583748 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:17.584198 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:17.585067 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:17.585270 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:17.585326 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:17.586558 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:52:17.586560 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:52:17.586560 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:52:17.586567 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:52:17.588331 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:52:17.588762 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:52:17.588762 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:52:17.588763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:52:17.588815 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:52:17.589005 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:52:17.589006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:52:17.589007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:52:17.589157 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:52:17.589265 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:17.589366 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 22:52:17.589428 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 22:52:17.591125 326 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55568:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:52:17.592803 326 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:52:17.595655 448 tablet_service.cc:1467] Processing CreateTablet for tablet 92c167ca01a14672879ff21a5f639e3a (DEFAULT_TABLE table=test-table [id=ab50401a996b48e7be180ba7e79529a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:52:17.595868 448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92c167ca01a14672879ff21a5f639e3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:52:17.597322 406 tablet_bootstrap.cc:492] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041: Bootstrap starting.
I20260928 22:52:17.597658 406 tablet_bootstrap.cc:654] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041: Neither blocks nor log segments found. Creating new log.
I20260928 22:52:17.598726 406 tablet_bootstrap.cc:492] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041: No bootstrap required, opened a new log
I20260928 22:52:17.598784 406 ts_tablet_manager.cc:1397] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 22:52:17.598928 406 raft_consensus.cc:348] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc5b62cd2534fd090ba8c61fdb3c421" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 33137 } } peers { permanent_uuid: "61f7571e3b354ccd803db55846d546d5" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 37823 } } peers { permanent_uuid: "64000c9c5f484909b35f01e352a21041" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37381 } }
I20260928 22:52:17.599017 406 raft_consensus.cc:374] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:52:17.599105 406 raft_consensus.cc:729] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64000c9c5f484909b35f01e352a21041, State: Initialized, Role: FOLLOWER
I20260928 22:52:17.599153 406 consensus_queue.cc:260] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc5b62cd2534fd090ba8c61fdb3c421" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 33137 } } peers { permanent_uuid: "61f7571e3b354ccd803db55846d546d5" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 37823 } } peers { permanent_uuid: "64000c9c5f484909b35f01e352a21041" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37381 } }
I20260928 22:52:17.599242 406 ts_tablet_manager.cc:1428] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 22:52:17.599424 521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:17.595081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.131:37823 (local address 127.0.0.1:57110)
0928 22:52:17.595140 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:17.595153 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:52:17.595204 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:52:17.595690 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:52:17.595691 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:52:17.595692 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:52:17.595746 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:17.595748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:17.597165 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:17.597166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:17.598153 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:17.598156 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:17.599131 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:17.599132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:17.599212 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:17.599246 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 22:52:17.599284 (+ 38us) client_negotiation.cc:240] Negotiation successful
0928 22:52:17.599333 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":2}
I20260928 22:52:17.599643 790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:17.595081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.130:33137 (local address 127.0.0.1:56804)
0928 22:52:17.595289 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:17.595298 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:52:17.595334 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:52:17.595675 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:52:17.595676 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:52:17.595679 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:52:17.595746 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:17.595748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:17.597195 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:17.597196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:17.598387 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:17.598391 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:17.599393 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:17.599394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:17.599457 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:17.599461 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:52:17.599525 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 22:52:17.599573 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":21,"thread_start_us":71,"threads_started":1}
I20260928 22:52:17.599678 792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:17.595312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56804 (local address 127.16.121.130:33137)
0928 22:52:17.595474 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:17.595480 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:17.595491 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:52:17.595537 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:17.595539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:17.595555 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:17.595640 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:52:17.595840 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:17.597095 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:17.598518 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:17.599302 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:17.599393 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:17.599473 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:17.599542 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 22:52:17.599594 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":88,"threads_started":1}
I20260928 22:52:17.599874 791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:17.595208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57110 (local address 127.16.121.131:37823)
0928 22:52:17.595428 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:17.595433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:17.595442 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:52:17.595550 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:17.595552 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:17.595565 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:17.595822 (+ 257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:52:17.595911 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:17.597036 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:17.598244 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:17.598990 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:17.599599 (+ 609us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:17.599680 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:17.599708 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 22:52:17.599769 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":94,"threads_started":1}
I20260928 22:52:17.600660 586 tablet_service.cc:1467] Processing CreateTablet for tablet 92c167ca01a14672879ff21a5f639e3a (DEFAULT_TABLE table=test-table [id=ab50401a996b48e7be180ba7e79529a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:52:17.600657 718 tablet_service.cc:1467] Processing CreateTablet for tablet 92c167ca01a14672879ff21a5f639e3a (DEFAULT_TABLE table=test-table [id=ab50401a996b48e7be180ba7e79529a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:52:17.601066 718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92c167ca01a14672879ff21a5f639e3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:52:17.601084 586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92c167ca01a14672879ff21a5f639e3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:52:17.604143 795 tablet_bootstrap.cc:492] T 92c167ca01a14672879ff21a5f639e3a P 61f7571e3b354ccd803db55846d546d5: Bootstrap starting.
I20260928 22:52:17.604815 795 tablet_bootstrap.cc:654] T 92c167ca01a14672879ff21a5f639e3a P 61f7571e3b354ccd803db55846d546d5: Neither blocks nor log segments found. Creating new log.
I20260928 22:52:17.605347 795 log.cc:824] T 92c167ca01a14672879ff21a5f639e3a P 61f7571e3b354ccd803db55846d546d5: Log is configured to *not* fsync() on all Append() calls
I20260928 22:52:17.605679 794 tablet_bootstrap.cc:492] T 92c167ca01a14672879ff21a5f639e3a P 6fc5b62cd2534fd090ba8c61fdb3c421: Bootstrap starting.
I20260928 22:52:17.606186 795 tablet_bootstrap.cc:492] T 92c167ca01a14672879ff21a5f639e3a P 61f7571e3b354ccd803db55846d546d5: No bootstrap required, opened a new log
I20260928 22:52:17.606246 795 ts_tablet_manager.cc:1397] T 92c167ca01a14672879ff21a5f639e3a P 61f7571e3b354ccd803db55846d546d5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:52:17.606257 794 tablet_bootstrap.cc:654] T 92c167ca01a14672879ff21a5f639e3a P 6fc5b62cd2534fd090ba8c61fdb3c421: Neither blocks nor log segments found. Creating new log.
I20260928 22:52:17.606707 794 log.cc:824] T 92c167ca01a14672879ff21a5f639e3a P 6fc5b62cd2534fd090ba8c61fdb3c421: Log is configured to *not* fsync() on all Append() calls
I20260928 22:52:17.607887 794 tablet_bootstrap.cc:492] T 92c167ca01a14672879ff21a5f639e3a P 6fc5b62cd2534fd090ba8c61fdb3c421: No bootstrap required, opened a new log
I20260928 22:52:17.607977 794 ts_tablet_manager.cc:1397] T 92c167ca01a14672879ff21a5f639e3a P 6fc5b62cd2534fd090ba8c61fdb3c421: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:52:17.608166 795 raft_consensus.cc:348] T 92c167ca01a14672879ff21a5f639e3a P 61f7571e3b354ccd803db55846d546d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc5b62cd2534fd090ba8c61fdb3c421" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 33137 } } peers { permanent_uuid: "61f7571e3b354ccd803db55846d546d5" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 37823 } } peers { permanent_uuid: "64000c9c5f484909b35f01e352a21041" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37381 } }
I20260928 22:52:17.608314 795 raft_consensus.cc:374] T 92c167ca01a14672879ff21a5f639e3a P 61f7571e3b354ccd803db55846d546d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:52:17.608335 795 raft_consensus.cc:729] T 92c167ca01a14672879ff21a5f639e3a P 61f7571e3b354ccd803db55846d546d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61f7571e3b354ccd803db55846d546d5, State: Initialized, Role: FOLLOWER
I20260928 22:52:17.608495 795 consensus_queue.cc:260] T 92c167ca01a14672879ff21a5f639e3a P 61f7571e3b354ccd803db55846d546d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc5b62cd2534fd090ba8c61fdb3c421" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 33137 } } peers { permanent_uuid: "61f7571e3b354ccd803db55846d546d5" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 37823 } } peers { permanent_uuid: "64000c9c5f484909b35f01e352a21041" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37381 } }
I20260928 22:52:17.608816 795 ts_tablet_manager.cc:1428] T 92c167ca01a14672879ff21a5f639e3a P 61f7571e3b354ccd803db55846d546d5: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 22:52:17.608835 784 heartbeater.cc:503] Master 127.16.121.190:34005 was elected leader, sending a full tablet report...
I20260928 22:52:17.610100 794 raft_consensus.cc:348] T 92c167ca01a14672879ff21a5f639e3a P 6fc5b62cd2534fd090ba8c61fdb3c421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc5b62cd2534fd090ba8c61fdb3c421" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 33137 } } peers { permanent_uuid: "61f7571e3b354ccd803db55846d546d5" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 37823 } } peers { permanent_uuid: "64000c9c5f484909b35f01e352a21041" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37381 } }
I20260928 22:52:17.610292 794 raft_consensus.cc:374] T 92c167ca01a14672879ff21a5f639e3a P 6fc5b62cd2534fd090ba8c61fdb3c421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:52:17.610342 794 raft_consensus.cc:729] T 92c167ca01a14672879ff21a5f639e3a P 6fc5b62cd2534fd090ba8c61fdb3c421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fc5b62cd2534fd090ba8c61fdb3c421, State: Initialized, Role: FOLLOWER
I20260928 22:52:17.610503 794 consensus_queue.cc:260] T 92c167ca01a14672879ff21a5f639e3a P 6fc5b62cd2534fd090ba8c61fdb3c421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc5b62cd2534fd090ba8c61fdb3c421" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 33137 } } peers { permanent_uuid: "61f7571e3b354ccd803db55846d546d5" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 37823 } } peers { permanent_uuid: "64000c9c5f484909b35f01e352a21041" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37381 } }
I20260928 22:52:17.610795 794 ts_tablet_manager.cc:1428] T 92c167ca01a14672879ff21a5f639e3a P 6fc5b62cd2534fd090ba8c61fdb3c421: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:52:17.610836 652 heartbeater.cc:503] Master 127.16.121.190:34005 was elected leader, sending a full tablet report...
W20260928 22:52:17.673529 653 tablet.cc:2369] T 92c167ca01a14672879ff21a5f639e3a P 6fc5b62cd2534fd090ba8c61fdb3c421: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:52:17.695886 520 raft_consensus.cc:482] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:52:17.696051 520 raft_consensus.cc:504] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc5b62cd2534fd090ba8c61fdb3c421" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 33137 } } peers { permanent_uuid: "61f7571e3b354ccd803db55846d546d5" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 37823 } } peers { permanent_uuid: "64000c9c5f484909b35f01e352a21041" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37381 } }
I20260928 22:52:17.696485 520 leader_election.cc:288] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fc5b62cd2534fd090ba8c61fdb3c421 (127.16.121.130:33137), 61f7571e3b354ccd803db55846d546d5 (127.16.121.131:37823)
I20260928 22:52:17.699867 792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:17.696999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:58087 (local address 127.16.121.130:33137)
0928 22:52:17.697042 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:17.697044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:17.697056 (+ 12us) server_negotiation.cc:407] Connection header received
0928 22:52:17.697088 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:17.697089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:17.697102 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:17.697167 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:52:17.697374 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:17.697982 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:17.698984 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:17.699462 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:17.699522 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:17.699529 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:17.699744 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 22:52:17.699790 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 22:52:17.699892 516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:17.696811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.130:33137 (local address 127.16.121.129:58087)
0928 22:52:17.696908 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:17.696923 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 22:52:17.696999 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:52:17.697208 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:52:17.697209 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:52:17.697212 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:52:17.697294 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:17.697297 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:17.698095 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:17.698096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:17.698888 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:17.698890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:17.699548 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:17.699549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:17.699648 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:17.699667 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 22:52:17.699711 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 22:52:17.699768 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 22:52:17.699929 791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:17.696972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:35185 (local address 127.16.121.131:37823)
0928 22:52:17.697023 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:17.697025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:17.697203 (+ 178us) server_negotiation.cc:407] Connection header received
0928 22:52:17.697232 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:17.697234 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:17.697243 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:17.697296 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:52:17.697518 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:17.697970 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:17.699038 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:17.699483 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:17.699577 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:17.699586 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:17.699782 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 22:52:17.699828 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 22:52:17.699905 800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:17.696812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.131:37823 (local address 127.16.121.129:35185)
0928 22:52:17.697082 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:17.697092 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:52:17.697158 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:52:17.697317 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:52:17.697317 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:52:17.697319 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:52:17.697420 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:17.697421 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:17.698063 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:17.698064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:17.698894 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:17.698897 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:17.699617 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:17.699618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:17.699683 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:17.699686 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:52:17.699762 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 22:52:17.699797 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":69,"thread_start_us":147,"threads_started":1}
I20260928 22:52:17.700299 606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c167ca01a14672879ff21a5f639e3a" candidate_uuid: "64000c9c5f484909b35f01e352a21041" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fc5b62cd2534fd090ba8c61fdb3c421" is_pre_election: true
I20260928 22:52:17.700371 738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c167ca01a14672879ff21a5f639e3a" candidate_uuid: "64000c9c5f484909b35f01e352a21041" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61f7571e3b354ccd803db55846d546d5" is_pre_election: true
I20260928 22:52:17.700476 606 raft_consensus.cc:2454] T 92c167ca01a14672879ff21a5f639e3a P 6fc5b62cd2534fd090ba8c61fdb3c421 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64000c9c5f484909b35f01e352a21041 in term 0.
I20260928 22:52:17.700595 738 raft_consensus.cc:2454] T 92c167ca01a14672879ff21a5f639e3a P 61f7571e3b354ccd803db55846d546d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64000c9c5f484909b35f01e352a21041 in term 0.
I20260928 22:52:17.700724 400 leader_election.cc:302] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64000c9c5f484909b35f01e352a21041, 6fc5b62cd2534fd090ba8c61fdb3c421; no voters:
I20260928 22:52:17.700848 520 raft_consensus.cc:2781] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:52:17.700903 520 raft_consensus.cc:482] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:52:17.700991 520 raft_consensus.cc:3037] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:52:17.701793 520 raft_consensus.cc:504] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc5b62cd2534fd090ba8c61fdb3c421" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 33137 } } peers { permanent_uuid: "61f7571e3b354ccd803db55846d546d5" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 37823 } } peers { permanent_uuid: "64000c9c5f484909b35f01e352a21041" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37381 } }
I20260928 22:52:17.701946 520 leader_election.cc:288] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [CANDIDATE]: Term 1 election: Requested vote from peers 6fc5b62cd2534fd090ba8c61fdb3c421 (127.16.121.130:33137), 61f7571e3b354ccd803db55846d546d5 (127.16.121.131:37823)
I20260928 22:52:17.702105 738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c167ca01a14672879ff21a5f639e3a" candidate_uuid: "64000c9c5f484909b35f01e352a21041" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61f7571e3b354ccd803db55846d546d5"
I20260928 22:52:17.702109 606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c167ca01a14672879ff21a5f639e3a" candidate_uuid: "64000c9c5f484909b35f01e352a21041" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fc5b62cd2534fd090ba8c61fdb3c421"
I20260928 22:52:17.702183 606 raft_consensus.cc:3037] T 92c167ca01a14672879ff21a5f639e3a P 6fc5b62cd2534fd090ba8c61fdb3c421 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:52:17.702203 738 raft_consensus.cc:3037] T 92c167ca01a14672879ff21a5f639e3a P 61f7571e3b354ccd803db55846d546d5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:52:17.702898 606 raft_consensus.cc:2454] T 92c167ca01a14672879ff21a5f639e3a P 6fc5b62cd2534fd090ba8c61fdb3c421 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64000c9c5f484909b35f01e352a21041 in term 1.
I20260928 22:52:17.702975 738 raft_consensus.cc:2454] T 92c167ca01a14672879ff21a5f639e3a P 61f7571e3b354ccd803db55846d546d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64000c9c5f484909b35f01e352a21041 in term 1.
I20260928 22:52:17.703094 400 leader_election.cc:302] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64000c9c5f484909b35f01e352a21041, 6fc5b62cd2534fd090ba8c61fdb3c421; no voters:
I20260928 22:52:17.703189 520 raft_consensus.cc:2781] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:52:17.703392 520 raft_consensus.cc:686] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [term 1 LEADER]: Becoming Leader. State: Replica: 64000c9c5f484909b35f01e352a21041, State: Running, Role: LEADER
I20260928 22:52:17.703475 520 consensus_queue.cc:237] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc5b62cd2534fd090ba8c61fdb3c421" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 33137 } } peers { permanent_uuid: "61f7571e3b354ccd803db55846d546d5" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 37823 } } peers { permanent_uuid: "64000c9c5f484909b35f01e352a21041" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37381 } }
I20260928 22:52:17.704289 325 catalog_manager.cc:5495] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64000c9c5f484909b35f01e352a21041 (127.16.121.129). New cstate: current_term: 1 leader_uuid: "64000c9c5f484909b35f01e352a21041" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc5b62cd2534fd090ba8c61fdb3c421" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 33137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61f7571e3b354ccd803db55846d546d5" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 37823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64000c9c5f484909b35f01e352a21041" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37381 } health_report { overall_health: HEALTHY } } }
I20260928 22:52:17.717200 522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:17.714627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44576 (local address 127.16.121.129:37381)
0928 22:52:17.714709 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:17.714711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:17.714722 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:52:17.714790 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:17.714791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:17.714809 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:17.714877 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:52:17.715148 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:17.715712 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:17.716431 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:17.716680 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:17.716746 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:17.716858 (+ 112us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:52:17.716964 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:52:17.717017 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:17.717054 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 22:52:17.717072 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 22:52:17.718824 738 raft_consensus.cc:1261] T 92c167ca01a14672879ff21a5f639e3a P 61f7571e3b354ccd803db55846d546d5 [term 1 FOLLOWER]: Refusing update from remote peer 64000c9c5f484909b35f01e352a21041: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:52:17.718824 606 raft_consensus.cc:1261] T 92c167ca01a14672879ff21a5f639e3a P 6fc5b62cd2534fd090ba8c61fdb3c421 [term 1 FOLLOWER]: Refusing update from remote peer 64000c9c5f484909b35f01e352a21041: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:52:17.719290 520 consensus_queue.cc:1035] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61f7571e3b354ccd803db55846d546d5" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 37823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:52:17.719478 801 consensus_queue.cc:1035] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fc5b62cd2534fd090ba8c61fdb3c421" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 33137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:52:17.722697 806 mvcc.cc:205] Tried to move back new op lower bound from 7334444800893550592 to 7334444800833937408. Current Snapshot: MvccSnapshot[applied={T|T < 7334444800893550592}]
I20260928 22:52:17.724016 325 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:52:17.724148 325 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:52:17.724905 325 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c: Sending DeleteTablet for 3 replicas of tablet 92c167ca01a14672879ff21a5f639e3a
I20260928 22:52:17.725251 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 364
I20260928 22:52:17.725242 718 tablet_service.cc:1514] Processing DeleteTablet for tablet 92c167ca01a14672879ff21a5f639e3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:52:17 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:57110
I20260928 22:52:17.726609 586 tablet_service.cc:1514] Processing DeleteTablet for tablet 92c167ca01a14672879ff21a5f639e3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:52:17 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:56804
I20260928 22:52:17.726927 815 tablet_replica.cc:330] T 92c167ca01a14672879ff21a5f639e3a P 6fc5b62cd2534fd090ba8c61fdb3c421: stopping tablet replica
I20260928 22:52:17.727072 815 raft_consensus.cc:2229] T 92c167ca01a14672879ff21a5f639e3a P 6fc5b62cd2534fd090ba8c61fdb3c421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:52:17.727185 815 raft_consensus.cc:2258] T 92c167ca01a14672879ff21a5f639e3a P 6fc5b62cd2534fd090ba8c61fdb3c421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:52:17.727648 815 ts_tablet_manager.cc:1905] T 92c167ca01a14672879ff21a5f639e3a P 6fc5b62cd2534fd090ba8c61fdb3c421: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:52:17.729382 815 ts_tablet_manager.cc:1918] T 92c167ca01a14672879ff21a5f639e3a P 6fc5b62cd2534fd090ba8c61fdb3c421: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:52:17.729472 815 log.cc:1196] T 92c167ca01a14672879ff21a5f639e3a P 6fc5b62cd2534fd090ba8c61fdb3c421: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-1/wal/wals/92c167ca01a14672879ff21a5f639e3a
I20260928 22:52:17.729789 815 ts_tablet_manager.cc:1939] T 92c167ca01a14672879ff21a5f639e3a P 6fc5b62cd2534fd090ba8c61fdb3c421: Deleting consensus metadata
I20260928 22:52:17.730278 310 catalog_manager.cc:4841] TS 6fc5b62cd2534fd090ba8c61fdb3c421 (127.16.121.130:33137): tablet 92c167ca01a14672879ff21a5f639e3a (table test-table [id=ab50401a996b48e7be180ba7e79529a6]) successfully deleted
I20260928 22:52:17.731195 814 tablet_replica.cc:330] T 92c167ca01a14672879ff21a5f639e3a P 61f7571e3b354ccd803db55846d546d5: stopping tablet replica
I20260928 22:52:17.731338 814 raft_consensus.cc:2229] T 92c167ca01a14672879ff21a5f639e3a P 61f7571e3b354ccd803db55846d546d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:52:17.731474 814 raft_consensus.cc:2258] T 92c167ca01a14672879ff21a5f639e3a P 61f7571e3b354ccd803db55846d546d5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 22:52:17.731465 310 connection.cc:535] client connection to 127.16.121.129:37381 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 22:52:17.731616 310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 22:52:17.731897 814 ts_tablet_manager.cc:1905] T 92c167ca01a14672879ff21a5f639e3a P 61f7571e3b354ccd803db55846d546d5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 22:52:17.732100 790 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:52:17.731984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:37381 (local address 127.0.0.1:44580)
0928 22:52:17.732026 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:17.732061 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.129:37381: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":20}
I20260928 22:52:17.732527 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 519
W20260928 22:52:17.732242 310 catalog_manager.cc:4584] TS 64000c9c5f484909b35f01e352a21041 (127.16.121.129:37381): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 92c167ca01a14672879ff21a5f639e3a: Network error: Client connection negotiation failed: client connection to 127.16.121.129:37381: connect: Connection reset by peer (error 104)
I20260928 22:52:17.735411 814 ts_tablet_manager.cc:1918] T 92c167ca01a14672879ff21a5f639e3a P 61f7571e3b354ccd803db55846d546d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:52:17.735496 814 log.cc:1196] T 92c167ca01a14672879ff21a5f639e3a P 61f7571e3b354ccd803db55846d546d5: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-2/wal/wals/92c167ca01a14672879ff21a5f639e3a
I20260928 22:52:17.735796 814 ts_tablet_manager.cc:1939] T 92c167ca01a14672879ff21a5f639e3a P 61f7571e3b354ccd803db55846d546d5: Deleting consensus metadata
I20260928 22:52:17.736565 312 catalog_manager.cc:4841] TS 61f7571e3b354ccd803db55846d546d5 (127.16.121.131:37823): tablet 92c167ca01a14672879ff21a5f639e3a (table test-table [id=ab50401a996b48e7be180ba7e79529a6]) successfully deleted
I20260928 22:52:17.738677 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 655
I20260928 22:52:17.744660 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 32761
I20260928 22:52:37.750394 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:34005
--webserver_interface=127.16.121.190
--webserver_port=37533
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:42335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:34005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:52:37.837136 817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:52:37.837385 817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:52:37.837423 817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:52:37.838708 817 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 22:52:37.838780 817 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 22:52:37.838801 817 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 22:52:37.838821 817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:52:37.838850 817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:52:37.838873 817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:52:37.838902 817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:52:37.838933 817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:52:37.840442 817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:42335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:34005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:34005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.121.190
--webserver_port=37533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:52:37.840721 817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:52:37.840947 817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:52:37.846072 823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:52:37.846086 822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:52:37.846072 825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:52:37.846302 817 server_base.cc:1034] running on GCE node
I20260928 22:52:37.846582 817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:52:37.846907 817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:52:37.848110 817 hybrid_clock.cc:648] HybridClock initialized: now 1790635957848087 us; error 45 us; skew 500 ppm
Sep 28 22:52:37 dist-test-slave-56vw krb5kdc[32213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635957, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:52:37.850909 817 init.cc:377] Logged in from keytab as kudu/127.16.121.190@KRBTEST.COM (short username kudu)
I20260928 22:52:37.851500 817 webserver.cc:460] Webserver started at http://127.16.121.190:37533/ using document root <none> and password file <none>
I20260928 22:52:37.851754 817 fs_manager.cc:359] Metadata directory not provided
I20260928 22:52:37.851822 817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:52:37.853281 817 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 22:52:37.854043 832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:37.854259 817 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:52:37.854367 817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "6c9fdbbae88b4180bcbb87227139de8c"
format_stamp: "Formatted at 2026-09-28 22:51:56 on dist-test-slave-56vw"
server_key: "6f6ce0f70f5a723c4b3876801326714f"
server_key_iv: "db908605c2ebfc5e870579b9e8dd415f"
server_key_version: "encryptionkey@0"
I20260928 22:52:37.854776 817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:52:37.864312 817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:52:37.866475 817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:52:37.866638 817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:52:37.870882 817 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.190:34005
I20260928 22:52:37.870914 884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.190:34005 every 8 connection(s)
I20260928 22:52:37.871357 817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:52:37.872844 885 sys_catalog.cc:263] Verifying existing consensus state
I20260928 22:52:37.873493 885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c: Bootstrap starting.
I20260928 22:52:37.875581 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 817
I20260928 22:52:37.875867 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.129:37381
--local_ip_for_outbound_sockets=127.16.121.129
--tserver_master_addrs=127.16.121.190:34005
--webserver_port=33913
--webserver_interface=127.16.121.129
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab
--principal=kudu/127.16.121.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:42335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:52:37.882275 885 log.cc:824] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c: Log is configured to *not* fsync() on all Append() calls
I20260928 22:52:37.884696 885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 22:52:37.885047 885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c: Bootstrap complete.
I20260928 22:52:37.887427 885 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c9fdbbae88b4180bcbb87227139de8c" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 34005 } }
I20260928 22:52:37.887681 885 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c9fdbbae88b4180bcbb87227139de8c, State: Initialized, Role: FOLLOWER
I20260928 22:52:37.887874 885 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c9fdbbae88b4180bcbb87227139de8c" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 34005 } }
I20260928 22:52:37.887953 885 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:52:37.888010 885 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:52:37.888064 885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [term 2 FOLLOWER]: Advancing to term 3
I20260928 22:52:37.889078 885 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c9fdbbae88b4180bcbb87227139de8c" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 34005 } }
I20260928 22:52:37.889204 885 leader_election.cc:302] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c9fdbbae88b4180bcbb87227139de8c; no voters:
I20260928 22:52:37.889425 885 leader_election.cc:288] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 22:52:37.889467 889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [term 3 FOLLOWER]: Leader election won for term 3
I20260928 22:52:37.889585 889 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [term 3 LEADER]: Becoming Leader. State: Replica: 6c9fdbbae88b4180bcbb87227139de8c, State: Running, Role: LEADER
I20260928 22:52:37.889705 885 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:52:37.889732 889 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c9fdbbae88b4180bcbb87227139de8c" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 34005 } }
I20260928 22:52:37.890076 890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "6c9fdbbae88b4180bcbb87227139de8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c9fdbbae88b4180bcbb87227139de8c" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 34005 } } }
I20260928 22:52:37.890261 890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [sys.catalog]: This master's current role is: LEADER
I20260928 22:52:37.890069 889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c9fdbbae88b4180bcbb87227139de8c. Latest consensus state: current_term: 3 leader_uuid: "6c9fdbbae88b4180bcbb87227139de8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c9fdbbae88b4180bcbb87227139de8c" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 34005 } } }
I20260928 22:52:37.890655 889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c [sys.catalog]: This master's current role is: LEADER
I20260928 22:52:37.890781 897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:52:37.891646 897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:52:37.891814 897 catalog_manager.cc:1259] Loaded cluster ID: c471a56f6b4143c9b6e4cfa2a6484473
I20260928 22:52:37.891999 897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:52:37.892795 897 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:52:37.892921 897 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c: Loaded TSK: 0
I20260928 22:52:37.892944 897 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6c9fdbbae88b4180bcbb87227139de8c: Loaded TSK: 1
F20260928 22:52:37.893098 897 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f8b161a99ad google::LogMessage::Fail() at ??:0
@ 0x7f8b161ac88a google::LogMessage::SendToLog() at ??:0
@ 0x7f8b161a94de google::LogMessage::Flush() at ??:0
@ 0x7f8b161aaa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f8b18916104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f8b18913d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f8b1e1031cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f8b1e10348e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f8b1e103594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f8b1e0c9c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f8b1e0d3ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f8b180770d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f8b1806c954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f8b18ed06db start_thread at ??:0
@ 0x7f8b14ca761f clone at ??:0
W20260928 22:52:37.994031 887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:52:37.994344 887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:52:37.994388 887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:52:37.996228 887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:52:37.996301 887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:52:37.996354 887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.129
I20260928 22:52:37.998703 887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:42335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab
--principal=kudu/127.16.121.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.129:37381
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.121.129
--webserver_port=33913
--tserver_master_addrs=127.16.121.190:34005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.129
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:52:37.998993 887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:52:37.999291 887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:52:38.005096 927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:52:38.005096 926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:52:38.005298 929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:52:38.006299 887 server_base.cc:1034] running on GCE node
I20260928 22:52:38.006582 887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:52:38.006878 887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:52:38.008062 887 hybrid_clock.cc:648] HybridClock initialized: now 1790635958008029 us; error 41 us; skew 500 ppm
Sep 28 22:52:38 dist-test-slave-56vw krb5kdc[32213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635958, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:52:38.011389 887 init.cc:377] Logged in from keytab as kudu/127.16.121.129@KRBTEST.COM (short username kudu)
I20260928 22:52:38.011839 887 webserver.cc:460] Webserver started at http://127.16.121.129:33913/ using document root <none> and password file <none>
I20260928 22:52:38.012094 887 fs_manager.cc:359] Metadata directory not provided
I20260928 22:52:38.012166 887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:52:38.013568 887 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:52:38.014353 936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:38.014567 887 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:52:38.014688 887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-0/wal
uuid: "64000c9c5f484909b35f01e352a21041"
format_stamp: "Formatted at 2026-09-28 22:51:56 on dist-test-slave-56vw"
server_key: "3979c4e8dcb7661137b395086cd04247"
server_key_iv: "9010955251a31455e4f4fbb931c618df"
server_key_version: "encryptionkey@0"
I20260928 22:52:38.015110 887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:52:38.031574 887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:52:38.034395 887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:52:38.034585 887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:52:38.034850 887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:52:38.035383 943 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 22:52:38.037477 887 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 22:52:38.037539 887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 22:52:38.037586 887 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 22:52:38.038264 887 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 22:52:38.038321 887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 22:52:38.038455 943 tablet_bootstrap.cc:492] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041: Bootstrap starting.
I20260928 22:52:38.045877 887 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.129:37381
I20260928 22:52:38.046276 887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
I20260928 22:52:38.046748 1050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.129:37381 every 8 connection(s)
I20260928 22:52:38.049958 943 log.cc:824] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041: Log is configured to *not* fsync() on all Append() calls
I20260928 22:52:38.051535 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 887
I20260928 22:52:38.051827 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.130:33137
--local_ip_for_outbound_sockets=127.16.121.130
--tserver_master_addrs=127.16.121.190:34005
--webserver_port=44219
--webserver_interface=127.16.121.130
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab
--principal=kudu/127.16.121.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:42335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:52:38.052874 943 tablet_bootstrap.cc:492] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 22:52:38.053200 943 tablet_bootstrap.cc:492] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041: Bootstrap complete.
I20260928 22:52:38.054441 943 ts_tablet_manager.cc:1397] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20260928 22:52:38.056046 943 raft_consensus.cc:348] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc5b62cd2534fd090ba8c61fdb3c421" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 33137 } } peers { permanent_uuid: "61f7571e3b354ccd803db55846d546d5" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 37823 } } peers { permanent_uuid: "64000c9c5f484909b35f01e352a21041" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37381 } }
I20260928 22:52:38.056438 943 raft_consensus.cc:729] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64000c9c5f484909b35f01e352a21041, State: Initialized, Role: FOLLOWER
I20260928 22:52:38.056573 943 consensus_queue.cc:260] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc5b62cd2534fd090ba8c61fdb3c421" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 33137 } } peers { permanent_uuid: "61f7571e3b354ccd803db55846d546d5" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 37823 } } peers { permanent_uuid: "64000c9c5f484909b35f01e352a21041" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37381 } }
I20260928 22:52:38.056856 943 ts_tablet_manager.cc:1428] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
W20260928 22:52:38.120199 1053 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:52:38.047029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.129:47951)
0928 22:52:38.047266 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:38.047310 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 22:52:38.047868 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:52:38.119978 (+ 72110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":121,"threads_started":1}
W20260928 22:52:38.120462 1051 heartbeater.cc:644] Failed to heartbeat to 127.16.121.190:34005 (0 consecutive failures): Network error: Failed to ping master at 127.16.121.190:34005: Client connection negotiation failed: client connection to 127.16.121.190:34005: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 22:52:38.120877 1053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:38.120750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.129:49041)
0928 22:52:38.120807 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:38.120857 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 22:52:38.121249 1053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:38.121165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.129:44801)
0928 22:52:38.121211 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:38.121234 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 22:52:38.121356 1051 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 22:52:38.137521 1055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:52:38.137825 1055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:52:38.137868 1055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:52:38.139266 1055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:52:38.139328 1055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:52:38.139374 1055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.130
I20260928 22:52:38.140897 1055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:42335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab
--principal=kudu/127.16.121.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.130:33137
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.121.130
--webserver_port=44219
--tserver_master_addrs=127.16.121.190:34005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.130
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:52:38.141171 1055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:52:38.141422 1055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:52:38.146720 1064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:52:38.146726 1062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:52:38.146704 1061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:52:38.146993 1055 server_base.cc:1034] running on GCE node
I20260928 22:52:38.147249 1055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:52:38.147544 1055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:52:38.148725 1055 hybrid_clock.cc:648] HybridClock initialized: now 1790635958148709 us; error 49 us; skew 500 ppm
Sep 28 22:52:38 dist-test-slave-56vw krb5kdc[32213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635958, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:52:38.151305 1055 init.cc:377] Logged in from keytab as kudu/127.16.121.130@KRBTEST.COM (short username kudu)
I20260928 22:52:38.151813 1055 webserver.cc:460] Webserver started at http://127.16.121.130:44219/ using document root <none> and password file <none>
I20260928 22:52:38.152047 1055 fs_manager.cc:359] Metadata directory not provided
I20260928 22:52:38.152114 1055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:52:38.153534 1055 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:52:38.154278 1071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:38.154520 1055 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:52:38.154625 1055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-1/wal
uuid: "6fc5b62cd2534fd090ba8c61fdb3c421"
format_stamp: "Formatted at 2026-09-28 22:51:56 on dist-test-slave-56vw"
server_key: "d2fbd6564ca0fa3f5638e31cb3fef99a"
server_key_iv: "94f56c3cbea3dd56d164cc52f9928faf"
server_key_version: "encryptionkey@0"
I20260928 22:52:38.154996 1055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:52:38.169119 1055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:52:38.171185 1055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:52:38.171358 1055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:52:38.171612 1055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:52:38.171928 1055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:52:38.172001 1055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:38.172045 1055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:52:38.172071 1055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:38.178257 1055 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.130:33137
I20260928 22:52:38.178310 1184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.130:33137 every 8 connection(s)
I20260928 22:52:38.178668 1055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
I20260928 22:52:38.179805 1187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:38.179419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.130:38165)
0928 22:52:38.179584 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:38.179658 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":68,"threads_started":1}
W20260928 22:52:38.180222 1185 heartbeater.cc:644] Failed to heartbeat to 127.16.121.190:34005 (0 consecutive failures): Network error: Failed to ping master at 127.16.121.190:34005: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
I20260928 22:52:38.180544 1187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:38.180444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.130:60863)
0928 22:52:38.180494 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:38.180519 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 22:52:38.180809 1187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:38.180727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.130:40989)
0928 22:52:38.180769 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:38.180791 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 22:52:38.180918 1185 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 22:52:38.187428 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 1055
I20260928 22:52:38.187674 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.131:37823
--local_ip_for_outbound_sockets=127.16.121.131
--tserver_master_addrs=127.16.121.190:34005
--webserver_port=35385
--webserver_interface=127.16.121.131
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab
--principal=kudu/127.16.121.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:42335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:52:38.270943 1188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:52:38.271229 1188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:52:38.271257 1188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:52:38.272466 1188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:52:38.272508 1188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:52:38.272540 1188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.131
I20260928 22:52:38.274031 1188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:42335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab
--principal=kudu/127.16.121.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.131:37823
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.121.131
--webserver_port=35385
--tserver_master_addrs=127.16.121.190:34005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.131
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:52:38.274282 1188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:52:38.274518 1188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:52:38.279592 1194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:52:38.279635 1193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:52:38.279608 1196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:52:38.279858 1188 server_base.cc:1034] running on GCE node
I20260928 22:52:38.280084 1188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:52:38.280342 1188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:52:38.281472 1188 hybrid_clock.cc:648] HybridClock initialized: now 1790635958281454 us; error 36 us; skew 500 ppm
Sep 28 22:52:38 dist-test-slave-56vw krb5kdc[32213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635958, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:52:38.284406 1188 init.cc:377] Logged in from keytab as kudu/127.16.121.131@KRBTEST.COM (short username kudu)
I20260928 22:52:38.284941 1188 webserver.cc:460] Webserver started at http://127.16.121.131:35385/ using document root <none> and password file <none>
I20260928 22:52:38.285183 1188 fs_manager.cc:359] Metadata directory not provided
I20260928 22:52:38.285254 1188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:52:38.286660 1188 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:52:38.287434 1203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:38.287621 1188 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:52:38.287726 1188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-2/wal
uuid: "61f7571e3b354ccd803db55846d546d5"
format_stamp: "Formatted at 2026-09-28 22:51:56 on dist-test-slave-56vw"
server_key: "327cf5621defd0a04990ef8b41443833"
server_key_iv: "09a34ed5d2210dfdfe704fa7da140c59"
server_key_version: "encryptionkey@0"
I20260928 22:52:38.288111 1188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:52:38.310465 1188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:52:38.312949 1188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:52:38.313063 1188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:52:38.313244 1188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:52:38.313580 1188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:52:38.313621 1188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:38.313668 1188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:52:38.313694 1188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:38.320118 1188 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.131:37823
I20260928 22:52:38.320168 1316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.131:37823 every 8 connection(s)
I20260928 22:52:38.320510 1188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
I20260928 22:52:38.321779 1319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:38.321196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.131:53117)
0928 22:52:38.321345 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:38.321424 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
W20260928 22:52:38.322096 1317 heartbeater.cc:644] Failed to heartbeat to 127.16.121.190:34005 (0 consecutive failures): Network error: Failed to ping master at 127.16.121.190:34005: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
I20260928 22:52:38.322443 1319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:38.322344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.131:39819)
0928 22:52:38.322395 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:38.322425 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 22:52:38.322786 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 1188
I20260928 22:52:38.322912 1319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:38.322777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.131:49247)
0928 22:52:38.322822 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:38.322870 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 22:52:38.323029 1317 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 22:52:39.122005 1321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:39.121713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.129:46465)
0928 22:52:39.121894 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:39.121939 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":86,"threads_started":1}
I20260928 22:52:39.181565 1322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:39.181282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.130:51427)
0928 22:52:39.181458 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:39.181516 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":94,"threads_started":1}
I20260928 22:52:39.323585 1323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:39.323386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.131:59375)
0928 22:52:39.323497 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:39.323533 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":58,"threads_started":1}
I20260928 22:52:39.551318 1324 raft_consensus.cc:482] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:52:39.551450 1324 raft_consensus.cc:504] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc5b62cd2534fd090ba8c61fdb3c421" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 33137 } } peers { permanent_uuid: "61f7571e3b354ccd803db55846d546d5" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 37823 } } peers { permanent_uuid: "64000c9c5f484909b35f01e352a21041" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37381 } }
I20260928 22:52:39.551872 1324 leader_election.cc:288] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6fc5b62cd2534fd090ba8c61fdb3c421 (127.16.121.130:33137), 61f7571e3b354ccd803db55846d546d5 (127.16.121.131:37823)
Sep 28 22:52:39 dist-test-slave-56vw krb5kdc[32213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635958, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for kudu/127.16.121.130@KRBTEST.COM
Sep 28 22:52:39 dist-test-slave-56vw krb5kdc[32213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635958, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for kudu/127.16.121.131@KRBTEST.COM
I20260928 22:52:39.562703 1321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:39.552042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.131:37823 (local address 127.16.121.129:53453)
0928 22:52:39.552103 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:39.552116 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:52:39.552171 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:52:39.553147 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:52:39.553150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:52:39.553574 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:52:39.553742 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:39.553744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:39.554825 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:39.554826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:39.555668 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:39.555670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:39.556325 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:39.556326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:39.556450 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:39.557091 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:52:39.557096 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:52:39.559219 (+ 2123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:52:39.561139 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:52:39.561140 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:52:39.561140 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:52:39.561360 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:52:39.561626 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:52:39.561627 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:52:39.561627 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:52:39.561754 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:52:39.562175 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:52:39.562176 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:52:39.562293 (+ 117us) client_negotiation.cc:769] Sending connection context
0928 22:52:39.562411 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 22:52:39.562603 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":1}
I20260928 22:52:39.562718 1327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:39.552206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:45883 (local address 127.16.121.130:33137)
0928 22:52:39.552362 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:39.552372 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:39.552407 (+ 35us) server_negotiation.cc:407] Connection header received
0928 22:52:39.552974 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:39.552978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:39.553029 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:39.553165 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:52:39.553900 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:39.554715 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:39.555812 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:39.556190 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:39.556343 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:39.558886 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:52:39.558890 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:52:39.558890 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:52:39.558964 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:52:39.560938 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:52:39.561462 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:52:39.561465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:52:39.561467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:52:39.561556 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:52:39.561831 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:52:39.561833 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:52:39.561833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:52:39.562075 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:52:39.562164 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:39.562461 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 22:52:39.562591 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":95,"threads_started":1}
I20260928 22:52:39.562731 1326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:39.552167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:53453 (local address 127.16.121.131:37823)
0928 22:52:39.552353 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:39.552372 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:39.552398 (+ 26us) server_negotiation.cc:407] Connection header received
0928 22:52:39.552896 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:39.552900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:39.552966 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:39.553117 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:52:39.553873 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:39.554718 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:39.555782 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:39.556193 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:39.556342 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:39.559338 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:52:39.559342 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:52:39.559342 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:52:39.559356 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:52:39.561046 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:52:39.561450 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:52:39.561453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:52:39.561453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:52:39.561553 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:52:39.561833 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:52:39.561834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:52:39.561834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:52:39.562077 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:52:39.562177 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:39.562464 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 22:52:39.562595 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":64,"threads_started":1}
I20260928 22:52:39.562827 1325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:39.552103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.130:33137 (local address 127.16.121.129:45883)
0928 22:52:39.552227 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:39.552238 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:52:39.552293 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:52:39.553153 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:52:39.553154 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:52:39.553555 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:52:39.553736 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:39.553742 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:39.554842 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:39.554845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:39.555668 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:39.555671 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:39.556321 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:39.556322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:39.556451 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:39.557064 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:52:39.557072 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:52:39.558785 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:52:39.561036 (+ 2251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:52:39.561044 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:52:39.561046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:52:39.561359 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:52:39.561632 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:52:39.561633 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:52:39.561634 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:52:39.561758 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:52:39.562151 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:52:39.562158 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:52:39.562298 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 22:52:39.562411 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 22:52:39.562685 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":185,"thread_start_us":52,"threads_started":1}
W20260928 22:52:39.564169 937 leader_election.cc:341] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6fc5b62cd2534fd090ba8c61fdb3c421 (127.16.121.130:33137): Not found: Tablet not found: 92c167ca01a14672879ff21a5f639e3a
W20260928 22:52:39.564275 939 leader_election.cc:341] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 61f7571e3b354ccd803db55846d546d5 (127.16.121.131:37823): Not found: Tablet not found: 92c167ca01a14672879ff21a5f639e3a
I20260928 22:52:39.564327 939 leader_election.cc:302] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64000c9c5f484909b35f01e352a21041; no voters: 61f7571e3b354ccd803db55846d546d5, 6fc5b62cd2534fd090ba8c61fdb3c421
I20260928 22:52:39.564409 1324 raft_consensus.cc:2726] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 22:52:40.123157 1328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:40.122788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.129:53461)
0928 22:52:40.122950 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:40.123004 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":64,"threads_started":1}
I20260928 22:52:40.182262 1329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:40.182044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.130:49987)
0928 22:52:40.182169 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:40.182215 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":56,"threads_started":1}
I20260928 22:52:40.324383 1330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:40.324134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.131:52199)
0928 22:52:40.324295 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:40.324338 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":62,"threads_started":1}
I20260928 22:52:41.124003 1331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:41.123751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.129:40543)
0928 22:52:41.123907 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:41.123954 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":66,"threads_started":1}
I20260928 22:52:41.182958 1332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:41.182730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.130:42123)
0928 22:52:41.182878 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:41.182914 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":72,"threads_started":1}
I20260928 22:52:41.325166 1333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:41.324894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.131:41913)
0928 22:52:41.325047 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:41.325093 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":64,"threads_started":1}
I20260928 22:52:41.758608 1334 raft_consensus.cc:482] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:52:41.758729 1334 raft_consensus.cc:504] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc5b62cd2534fd090ba8c61fdb3c421" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 33137 } } peers { permanent_uuid: "61f7571e3b354ccd803db55846d546d5" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 37823 } } peers { permanent_uuid: "64000c9c5f484909b35f01e352a21041" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37381 } }
I20260928 22:52:41.758953 1334 leader_election.cc:288] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6fc5b62cd2534fd090ba8c61fdb3c421 (127.16.121.130:33137), 61f7571e3b354ccd803db55846d546d5 (127.16.121.131:37823)
W20260928 22:52:41.759486 937 leader_election.cc:341] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6fc5b62cd2534fd090ba8c61fdb3c421 (127.16.121.130:33137): Not found: Tablet not found: 92c167ca01a14672879ff21a5f639e3a
W20260928 22:52:41.759620 939 leader_election.cc:341] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 61f7571e3b354ccd803db55846d546d5 (127.16.121.131:37823): Not found: Tablet not found: 92c167ca01a14672879ff21a5f639e3a
I20260928 22:52:41.759730 939 leader_election.cc:302] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64000c9c5f484909b35f01e352a21041; no voters: 61f7571e3b354ccd803db55846d546d5, 6fc5b62cd2534fd090ba8c61fdb3c421
I20260928 22:52:41.759801 1334 raft_consensus.cc:2726] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 22:52:42.124831 1335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:42.124562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.129:52257)
0928 22:52:42.124737 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:42.124783 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":84,"threads_started":1}
I20260928 22:52:42.183700 1336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:42.183450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.130:37789)
0928 22:52:42.183602 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:42.183641 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":79,"threads_started":1}
I20260928 22:52:42.326079 1337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:42.325732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.131:33339)
0928 22:52:42.325897 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:42.325951 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":67,"threads_started":1}
I20260928 22:52:43.125624 1338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:43.125386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.129:45913)
0928 22:52:43.125532 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:43.125577 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":58,"threads_started":1}
I20260928 22:52:43.184412 1339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:43.184152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.130:58587)
0928 22:52:43.184308 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:43.184365 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":65,"threads_started":1}
I20260928 22:52:43.326875 1340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:43.326622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.131:44461)
0928 22:52:43.326781 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:43.326828 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":63,"threads_started":1}
I20260928 22:52:44.126444 1341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:44.126183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.129:43649)
0928 22:52:44.126346 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:44.126393 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":85,"threads_started":1}
I20260928 22:52:44.185114 1342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:44.184891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.130:34415)
0928 22:52:44.185027 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:44.185067 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":61,"threads_started":1}
I20260928 22:52:44.327733 1343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:44.327440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.131:46237)
0928 22:52:44.327601 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:44.327647 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":64,"threads_started":1}
I20260928 22:52:44.489810 1344 raft_consensus.cc:482] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:52:44.489938 1344 raft_consensus.cc:504] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc5b62cd2534fd090ba8c61fdb3c421" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 33137 } } peers { permanent_uuid: "61f7571e3b354ccd803db55846d546d5" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 37823 } } peers { permanent_uuid: "64000c9c5f484909b35f01e352a21041" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37381 } }
I20260928 22:52:44.490172 1344 leader_election.cc:288] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6fc5b62cd2534fd090ba8c61fdb3c421 (127.16.121.130:33137), 61f7571e3b354ccd803db55846d546d5 (127.16.121.131:37823)
W20260928 22:52:44.490588 939 leader_election.cc:341] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 61f7571e3b354ccd803db55846d546d5 (127.16.121.131:37823): Not found: Tablet not found: 92c167ca01a14672879ff21a5f639e3a
W20260928 22:52:44.490690 937 leader_election.cc:341] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6fc5b62cd2534fd090ba8c61fdb3c421 (127.16.121.130:33137): Not found: Tablet not found: 92c167ca01a14672879ff21a5f639e3a
I20260928 22:52:44.490788 937 leader_election.cc:302] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64000c9c5f484909b35f01e352a21041; no voters: 61f7571e3b354ccd803db55846d546d5, 6fc5b62cd2534fd090ba8c61fdb3c421
I20260928 22:52:44.490869 1344 raft_consensus.cc:2726] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 22:52:45.127314 1345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:45.127096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.129:53123)
0928 22:52:45.127230 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:45.127269 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
I20260928 22:52:45.185936 1346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:45.185675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.130:41955)
0928 22:52:45.185829 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:45.185886 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":66,"threads_started":1}
I20260928 22:52:45.328511 1347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:45.328282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.131:43851)
0928 22:52:45.328414 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:45.328461 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":62,"threads_started":1}
I20260928 22:52:46.128337 1348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:46.128033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.129:40497)
0928 22:52:46.128215 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:46.128275 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":73,"threads_started":1}
I20260928 22:52:46.186874 1349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:46.186582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.130:32821)
0928 22:52:46.186748 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:46.186819 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":63,"threads_started":1}
I20260928 22:52:46.226468 1350 raft_consensus.cc:482] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:52:46.226608 1350 raft_consensus.cc:504] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc5b62cd2534fd090ba8c61fdb3c421" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 33137 } } peers { permanent_uuid: "61f7571e3b354ccd803db55846d546d5" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 37823 } } peers { permanent_uuid: "64000c9c5f484909b35f01e352a21041" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37381 } }
I20260928 22:52:46.226851 1350 leader_election.cc:288] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6fc5b62cd2534fd090ba8c61fdb3c421 (127.16.121.130:33137), 61f7571e3b354ccd803db55846d546d5 (127.16.121.131:37823)
W20260928 22:52:46.227285 937 leader_election.cc:341] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6fc5b62cd2534fd090ba8c61fdb3c421 (127.16.121.130:33137): Not found: Tablet not found: 92c167ca01a14672879ff21a5f639e3a
W20260928 22:52:46.227388 939 leader_election.cc:341] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 61f7571e3b354ccd803db55846d546d5 (127.16.121.131:37823): Not found: Tablet not found: 92c167ca01a14672879ff21a5f639e3a
I20260928 22:52:46.227456 939 leader_election.cc:302] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64000c9c5f484909b35f01e352a21041; no voters: 61f7571e3b354ccd803db55846d546d5, 6fc5b62cd2534fd090ba8c61fdb3c421
I20260928 22:52:46.227550 1350 raft_consensus.cc:2726] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 22:52:46.329289 1351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:46.329061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.131:50751)
0928 22:52:46.329194 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:46.329241 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":64,"threads_started":1}
I20260928 22:52:47.129168 1352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:47.128891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.129:50761)
0928 22:52:47.129035 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:47.129081 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":68,"threads_started":1}
I20260928 22:52:47.187609 1353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:47.187380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.130:36155)
0928 22:52:47.187527 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:47.187562 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":78,"threads_started":1}
I20260928 22:52:47.330077 1354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:47.329837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.131:38759)
0928 22:52:47.329981 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:47.330030 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":63,"threads_started":1}
I20260928 22:52:48.130108 1355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:48.129817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.129:57759)
0928 22:52:48.130010 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:48.130061 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":78,"threads_started":1}
I20260928 22:52:48.188472 1356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:48.188210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.130:46301)
0928 22:52:48.188376 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:48.188420 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":89,"threads_started":1}
I20260928 22:52:48.330972 1357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:48.330650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.131:36217)
0928 22:52:48.330855 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:48.330921 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":78,"threads_started":1}
I20260928 22:52:49.131028 1358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:49.130802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.129:45147)
0928 22:52:49.130937 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:49.130980 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":68,"threads_started":1}
I20260928 22:52:49.189391 1359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:49.189134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.130:37085)
0928 22:52:49.189283 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:49.189331 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":71,"threads_started":1}
I20260928 22:52:49.331878 1360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:49.331623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.131:34007)
0928 22:52:49.331782 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:49.331829 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":61,"threads_started":1}
I20260928 22:52:50.131819 1361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:50.131589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.129:38825)
0928 22:52:50.131726 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:50.131771 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":70,"threads_started":1}
I20260928 22:52:50.190243 1362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:50.189996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.130:45453)
0928 22:52:50.190160 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:50.190195 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":89,"threads_started":1}
I20260928 22:52:50.332780 1363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:50.332425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.131:36199)
0928 22:52:50.332620 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:50.332666 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":82,"threads_started":1}
I20260928 22:52:51.132642 1364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:51.132391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.129:56209)
0928 22:52:51.132538 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:51.132586 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":64,"threads_started":1}
I20260928 22:52:51.191018 1365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:51.190810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.130:39777)
0928 22:52:51.190934 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:51.190970 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":63,"threads_started":1}
I20260928 22:52:51.333562 1366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:51.333333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.131:59461)
0928 22:52:51.333468 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:51.333515 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":65,"threads_started":1}
I20260928 22:52:52.133740 1367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:52.133439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.129:56287)
0928 22:52:52.133622 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:52.133677 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":78,"threads_started":1}
I20260928 22:52:52.191977 1368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:52.191587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.130:49703)
0928 22:52:52.191847 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:52.191908 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":146,"threads_started":1}
I20260928 22:52:52.267590 1369 raft_consensus.cc:482] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:52:52.267750 1369 raft_consensus.cc:504] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fc5b62cd2534fd090ba8c61fdb3c421" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 33137 } } peers { permanent_uuid: "61f7571e3b354ccd803db55846d546d5" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 37823 } } peers { permanent_uuid: "64000c9c5f484909b35f01e352a21041" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 37381 } }
I20260928 22:52:52.267994 1369 leader_election.cc:288] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6fc5b62cd2534fd090ba8c61fdb3c421 (127.16.121.130:33137), 61f7571e3b354ccd803db55846d546d5 (127.16.121.131:37823)
W20260928 22:52:52.268469 939 leader_election.cc:341] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 61f7571e3b354ccd803db55846d546d5 (127.16.121.131:37823): Not found: Tablet not found: 92c167ca01a14672879ff21a5f639e3a
W20260928 22:52:52.268548 937 leader_election.cc:341] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6fc5b62cd2534fd090ba8c61fdb3c421 (127.16.121.130:33137): Not found: Tablet not found: 92c167ca01a14672879ff21a5f639e3a
I20260928 22:52:52.268636 937 leader_election.cc:302] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64000c9c5f484909b35f01e352a21041; no voters: 61f7571e3b354ccd803db55846d546d5, 6fc5b62cd2534fd090ba8c61fdb3c421
I20260928 22:52:52.268723 1369 raft_consensus.cc:2726] T 92c167ca01a14672879ff21a5f639e3a P 64000c9c5f484909b35f01e352a21041 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 22:52:52.334419 1370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:52.334154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.131:45629)
0928 22:52:52.334323 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:52.334370 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":80,"threads_started":1}
I20260928 22:52:53.134590 1371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:53.134367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.129:33453)
0928 22:52:53.134495 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:53.134531 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":62,"threads_started":1}
I20260928 22:52:53.192893 1372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:53.192642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.130:33365)
0928 22:52:53.192794 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:53.192845 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":78,"threads_started":1}
I20260928 22:52:53.325765 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 887
I20260928 22:52:53.331362 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 1055
I20260928 22:52:53.335299 1373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:53.335035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:34005 (local address 127.16.121.131:50015)
0928 22:52:53.335209 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:53.335254 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.190:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":115,"threads_started":1}
I20260928 22:52:53.336141 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 1188
2026-09-28T22:52:53Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58410 ms)
[----------] 30 tests from SecurityITest (240016 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:52:53 dist-test-slave-56vw krb5kdc[1376](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:52:53 dist-test-slave-56vw krb5kdc[1376](info): set up 2 sockets
Sep 28 22:52:53 dist-test-slave-56vw krb5kdc[1376](info): commencing operation
krb5kdc: starting...
W20260928 22:52:54.470875 16870 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Sep 28 22:52:54 dist-test-slave-56vw krb5kdc[1376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:52:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:52:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.121.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
I20260928 22:52:54.598127 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:42523
--webserver_interface=127.16.121.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:39733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:42523
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:52:54.681488 1389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:52:54.681725 1389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:52:54.681749 1389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:52:54.682886 1389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:52:54.682924 1389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:52:54.682937 1389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:52:54.682950 1389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:52:54.682962 1389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:52:54.684428 1389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:39733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:42523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:42523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.121.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:52:54.684624 1389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:52:54.684810 1389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:52:54.689735 1394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:52:54.689805 1395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:52:54.689755 1397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:52:54.689847 1389 server_base.cc:1034] running on GCE node
I20260928 22:52:54.690311 1389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:52:54.690586 1389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:52:54.691766 1389 hybrid_clock.cc:648] HybridClock initialized: now 1790635974691750 us; error 42 us; skew 500 ppm
Sep 28 22:52:54 dist-test-slave-56vw krb5kdc[1376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635974, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:52:54.694375 1389 init.cc:377] Logged in from keytab as kudu/127.16.121.190@KRBTEST.COM (short username kudu)
I20260928 22:52:54.694819 1389 webserver.cc:460] Webserver started at http://127.16.121.190:42625/ using document root <none> and password file <none>
I20260928 22:52:54.695114 1389 fs_manager.cc:359] Metadata directory not provided
I20260928 22:52:54.695184 1389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:52:54.695354 1389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:52:54.696475 1389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "f9ab10c1eb79469bbf5abb53dd2284c0"
format_stamp: "Formatted at 2026-09-28 22:52:54 on dist-test-slave-56vw"
server_key: "eaa4e6133855c60c3462a410e225635a"
server_key_iv: "afd8f83e8e944ebbc9a22a796ec3cf68"
server_key_version: "encryptionkey@0"
I20260928 22:52:54.696856 1389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "f9ab10c1eb79469bbf5abb53dd2284c0"
format_stamp: "Formatted at 2026-09-28 22:52:54 on dist-test-slave-56vw"
server_key: "eaa4e6133855c60c3462a410e225635a"
server_key_iv: "afd8f83e8e944ebbc9a22a796ec3cf68"
server_key_version: "encryptionkey@0"
I20260928 22:52:54.698320 1389 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:52:54.699177 1404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:54.699380 1389 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:52:54.699481 1389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "f9ab10c1eb79469bbf5abb53dd2284c0"
format_stamp: "Formatted at 2026-09-28 22:52:54 on dist-test-slave-56vw"
server_key: "eaa4e6133855c60c3462a410e225635a"
server_key_iv: "afd8f83e8e944ebbc9a22a796ec3cf68"
server_key_version: "encryptionkey@0"
I20260928 22:52:54.699579 1389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:52:54.722849 1389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:52:54.725359 1389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:52:54.725553 1389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:52:54.729735 1389 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.190:42523
I20260928 22:52:54.729787 1456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.190:42523 every 8 connection(s)
I20260928 22:52:54.730185 1389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:52:54.731132 1457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:52:54.733538 1457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9ab10c1eb79469bbf5abb53dd2284c0: Bootstrap starting.
I20260928 22:52:54.733552 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 1389
I20260928 22:52:54.733680 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:52:54.733883 16870 external_mini_cluster.cc:1428] Setting key c08ecc39127fec261e488e3ac80f4970
I20260928 22:52:54.734277 1457 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9ab10c1eb79469bbf5abb53dd2284c0: Neither blocks nor log segments found. Creating new log.
I20260928 22:52:54.734649 1457 log.cc:824] T 00000000000000000000000000000000 P f9ab10c1eb79469bbf5abb53dd2284c0: Log is configured to *not* fsync() on all Append() calls
I20260928 22:52:54.735577 1457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9ab10c1eb79469bbf5abb53dd2284c0: No bootstrap required, opened a new log
I20260928 22:52:54.737941 1457 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9ab10c1eb79469bbf5abb53dd2284c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9ab10c1eb79469bbf5abb53dd2284c0" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 42523 } }
I20260928 22:52:54.738117 1457 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9ab10c1eb79469bbf5abb53dd2284c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:52:54.738144 1457 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9ab10c1eb79469bbf5abb53dd2284c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9ab10c1eb79469bbf5abb53dd2284c0, State: Initialized, Role: FOLLOWER
I20260928 22:52:54.738282 1457 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9ab10c1eb79469bbf5abb53dd2284c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9ab10c1eb79469bbf5abb53dd2284c0" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 42523 } }
I20260928 22:52:54.738358 1457 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9ab10c1eb79469bbf5abb53dd2284c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:52:54.738421 1457 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9ab10c1eb79469bbf5abb53dd2284c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:52:54.738484 1457 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9ab10c1eb79469bbf5abb53dd2284c0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:52:54.739233 1457 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9ab10c1eb79469bbf5abb53dd2284c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9ab10c1eb79469bbf5abb53dd2284c0" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 42523 } }
I20260928 22:52:54.739344 1457 leader_election.cc:302] T 00000000000000000000000000000000 P f9ab10c1eb79469bbf5abb53dd2284c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9ab10c1eb79469bbf5abb53dd2284c0; no voters:
I20260928 22:52:54.739521 1457 leader_election.cc:288] T 00000000000000000000000000000000 P f9ab10c1eb79469bbf5abb53dd2284c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:52:54.739579 1462 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9ab10c1eb79469bbf5abb53dd2284c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:52:54.739797 1462 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9ab10c1eb79469bbf5abb53dd2284c0 [term 1 LEADER]: Becoming Leader. State: Replica: f9ab10c1eb79469bbf5abb53dd2284c0, State: Running, Role: LEADER
I20260928 22:52:54.739809 1457 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9ab10c1eb79469bbf5abb53dd2284c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:52:54.740608 1462 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9ab10c1eb79469bbf5abb53dd2284c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9ab10c1eb79469bbf5abb53dd2284c0" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 42523 } }
I20260928 22:52:54.741338 1463 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9ab10c1eb79469bbf5abb53dd2284c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9ab10c1eb79469bbf5abb53dd2284c0. Latest consensus state: current_term: 1 leader_uuid: "f9ab10c1eb79469bbf5abb53dd2284c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9ab10c1eb79469bbf5abb53dd2284c0" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 42523 } } }
I20260928 22:52:54.741385 1464 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9ab10c1eb79469bbf5abb53dd2284c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9ab10c1eb79469bbf5abb53dd2284c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9ab10c1eb79469bbf5abb53dd2284c0" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 42523 } } }
I20260928 22:52:54.741631 1463 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9ab10c1eb79469bbf5abb53dd2284c0 [sys.catalog]: This master's current role is: LEADER
I20260928 22:52:54.741637 1464 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9ab10c1eb79469bbf5abb53dd2284c0 [sys.catalog]: This master's current role is: LEADER
W20260928 22:52:54.741819 1476 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f9ab10c1eb79469bbf5abb53dd2284c0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:52:54.741868 1476 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:52:54.741923 1478 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:52:54.742094 1478 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 22:52:54 dist-test-slave-56vw krb5kdc[1376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:52:54.743804 1478 catalog_manager.cc:1347] Generated new cluster ID: 525613ce3fe94ed39e22cd79d7425ccc
I20260928 22:52:54.743860 1478 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:52:54.746752 1460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:54.734480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51722 (local address 127.16.121.190:42523)
0928 22:52:54.734695 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:54.734704 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:54.734743 (+ 39us) server_negotiation.cc:407] Connection header received
0928 22:52:54.735348 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:54.735358 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:54.735426 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:54.735584 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:52:54.736305 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:54.737831 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:54.738866 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:54.739561 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:54.739690 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:54.742846 (+ 3156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:52:54.742851 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:52:54.742852 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:52:54.742866 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:52:54.744999 (+ 2133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:52:54.745424 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:52:54.745427 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:52:54.745428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:52:54.745539 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:52:54.745765 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:52:54.745766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:52:54.745766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:52:54.746082 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:52:54.746236 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:54.746347 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 22:52:54.746496 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":89,"threads_started":1}
I20260928 22:52:54.751492 1478 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:52:54.752261 1478 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:52:54.757464 1478 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9ab10c1eb79469bbf5abb53dd2284c0: Generated new TSK 0
I20260928 22:52:54.757668 1478 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.121.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab.
Entry for principal kudu/127.16.121.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab.
I20260928 22:52:54.786640 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.129:0
--local_ip_for_outbound_sockets=127.16.121.129
--webserver_interface=127.16.121.129
--webserver_port=0
--tserver_master_addrs=127.16.121.190:42523
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab
--principal=kudu/127.16.121.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:39733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:52:54.884585 1483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:52:54.884820 1483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:52:54.884862 1483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:52:54.886188 1483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:52:54.886257 1483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:52:54.886303 1483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.129
I20260928 22:52:54.887900 1483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:39733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab
--principal=kudu/127.16.121.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.121.129
--webserver_port=0
--tserver_master_addrs=127.16.121.190:42523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.129
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:52:54.888173 1483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:52:54.888429 1483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:52:54.894434 1489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:52:54.894460 1491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:52:54.896310 1483 server_base.cc:1034] running on GCE node
W20260928 22:52:54.897759 1488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:52:54.898046 1483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:52:54.898373 1483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:52:54.899549 1483 hybrid_clock.cc:648] HybridClock initialized: now 1790635974899533 us; error 28 us; skew 500 ppm
Sep 28 22:52:54 dist-test-slave-56vw krb5kdc[1376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635974, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:52:54.903031 1483 init.cc:377] Logged in from keytab as kudu/127.16.121.129@KRBTEST.COM (short username kudu)
I20260928 22:52:54.903551 1483 webserver.cc:460] Webserver started at http://127.16.121.129:36975/ using document root <none> and password file <none>
I20260928 22:52:54.903785 1483 fs_manager.cc:359] Metadata directory not provided
I20260928 22:52:54.903831 1483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:52:54.903926 1483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:52:54.904944 1483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-0/data/instance:
uuid: "cc78e47f1b9c4c1ca1ab9f43905781af"
format_stamp: "Formatted at 2026-09-28 22:52:54 on dist-test-slave-56vw"
server_key: "80ec2483d6c455472fedb661429f658e"
server_key_iv: "024fe8916dbe4208e4e6cab395137ed2"
server_key_version: "encryptionkey@0"
I20260928 22:52:54.905236 1483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance:
uuid: "cc78e47f1b9c4c1ca1ab9f43905781af"
format_stamp: "Formatted at 2026-09-28 22:52:54 on dist-test-slave-56vw"
server_key: "80ec2483d6c455472fedb661429f658e"
server_key_iv: "024fe8916dbe4208e4e6cab395137ed2"
server_key_version: "encryptionkey@0"
I20260928 22:52:54.906635 1483 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 22:52:54.907496 1498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:54.907696 1483 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 22:52:54.907785 1483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-0/wal
uuid: "cc78e47f1b9c4c1ca1ab9f43905781af"
format_stamp: "Formatted at 2026-09-28 22:52:54 on dist-test-slave-56vw"
server_key: "80ec2483d6c455472fedb661429f658e"
server_key_iv: "024fe8916dbe4208e4e6cab395137ed2"
server_key_version: "encryptionkey@0"
I20260928 22:52:54.907841 1483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:52:54.922111 1483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:52:54.924733 1483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:52:54.924860 1483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:52:54.925071 1483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:52:54.925416 1483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:52:54.925458 1483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:54.925484 1483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:52:54.925498 1483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:54.931902 1483 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.129:33327
I20260928 22:52:54.931942 1611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.129:33327 every 8 connection(s)
I20260928 22:52:54.932349 1483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
I20260928 22:52:54.935972 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 1483
I20260928 22:52:54.936111 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance
I20260928 22:52:54.936332 16870 external_mini_cluster.cc:1428] Setting key aac60ea9fcee7f6d05c79c4b68b54fa4
Sep 28 22:52:54 dist-test-slave-56vw krb5kdc[1376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635974, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:52:54.944427 1460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:54.933218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:56019 (local address 127.16.121.190:42523)
0928 22:52:54.933352 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:54.933355 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:54.933550 (+ 195us) server_negotiation.cc:407] Connection header received
0928 22:52:54.934168 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:54.934170 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:54.934183 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:54.934244 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:52:54.935498 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:54.936315 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:54.937700 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:54.938110 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:54.938191 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:54.940777 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:52:54.940778 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:52:54.940779 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:52:54.940788 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:52:54.942683 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:52:54.943212 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:52:54.943214 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:52:54.943214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:52:54.943299 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:52:54.943627 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:52:54.943629 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:52:54.943629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:52:54.943813 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:52:54.943898 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:54.944188 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 22:52:54.944305 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260928 22:52:54.944456 1614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:54.933126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:42523 (local address 127.16.121.129:56019)
0928 22:52:54.933390 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:54.933442 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 22:52:54.934047 (+ 605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:52:54.934340 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:52:54.934343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:52:54.935158 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:52:54.935381 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:54.935386 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:54.936451 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:54.936453 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:54.937592 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:54.937596 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:54.938224 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:54.938225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:54.938346 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:54.938998 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:52:54.939008 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:52:54.940653 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:52:54.942778 (+ 2125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:52:54.942786 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:52:54.942788 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:52:54.943108 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:52:54.943404 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:52:54.943404 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:52:54.943405 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:52:54.943535 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:52:54.943890 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:52:54.943896 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:52:54.944026 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 22:52:54.944112 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 22:52:54.944271 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":92,"threads_started":1}
I20260928 22:52:54.945075 1612 heartbeater.cc:347] Connected to a master server at 127.16.121.190:42523
I20260928 22:52:54.945206 1612 heartbeater.cc:464] Registering TS with master...
I20260928 22:52:54.945434 1612 heartbeater.cc:511] Master 127.16.121.190:42523 requested a full tablet report, sending...
I20260928 22:52:54.946007 1421 ts_manager.cc:194] Registered new tserver with Master: cc78e47f1b9c4c1ca1ab9f43905781af (127.16.121.129:33327)
I20260928 22:52:54.946770 1421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.129@KRBTEST.COM'} at 127.16.121.129:56019
WARNING: no policy specified for kudu/127.16.121.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab.
Entry for principal kudu/127.16.121.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab.
I20260928 22:52:54.965879 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.130:0
--local_ip_for_outbound_sockets=127.16.121.130
--webserver_interface=127.16.121.130
--webserver_port=0
--tserver_master_addrs=127.16.121.190:42523
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab
--principal=kudu/127.16.121.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:39733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:52:55.047137 1617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:52:55.047351 1617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:52:55.047375 1617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:52:55.048456 1617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:52:55.048491 1617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:52:55.048516 1617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.130
I20260928 22:52:55.049861 1617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:39733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab
--principal=kudu/127.16.121.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.121.130
--webserver_port=0
--tserver_master_addrs=127.16.121.190:42523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.130
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:52:55.050040 1617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:52:55.050215 1617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:52:55.055348 1623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:52:55.055330 1625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:52:55.055352 1622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:52:55.055558 1617 server_base.cc:1034] running on GCE node
I20260928 22:52:55.055788 1617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:52:55.055996 1617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:52:55.057150 1617 hybrid_clock.cc:648] HybridClock initialized: now 1790635975057121 us; error 29 us; skew 500 ppm
Sep 28 22:52:55 dist-test-slave-56vw krb5kdc[1376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635975, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:52:55.059908 1617 init.cc:377] Logged in from keytab as kudu/127.16.121.130@KRBTEST.COM (short username kudu)
I20260928 22:52:55.060421 1617 webserver.cc:460] Webserver started at http://127.16.121.130:37937/ using document root <none> and password file <none>
I20260928 22:52:55.060667 1617 fs_manager.cc:359] Metadata directory not provided
I20260928 22:52:55.060711 1617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:52:55.060851 1617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:52:55.061978 1617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-1/data/instance:
uuid: "dbfa190872714f7596bda8ecf1f2801d"
format_stamp: "Formatted at 2026-09-28 22:52:55 on dist-test-slave-56vw"
server_key: "c0b57de046599f40f16637849648f4e7"
server_key_iv: "6750e7b40ba7ec097a748b2ed13d3522"
server_key_version: "encryptionkey@0"
I20260928 22:52:55.062351 1617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance:
uuid: "dbfa190872714f7596bda8ecf1f2801d"
format_stamp: "Formatted at 2026-09-28 22:52:55 on dist-test-slave-56vw"
server_key: "c0b57de046599f40f16637849648f4e7"
server_key_iv: "6750e7b40ba7ec097a748b2ed13d3522"
server_key_version: "encryptionkey@0"
I20260928 22:52:55.063831 1617 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:52:55.064705 1632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:55.064963 1617 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:52:55.065097 1617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-1/wal
uuid: "dbfa190872714f7596bda8ecf1f2801d"
format_stamp: "Formatted at 2026-09-28 22:52:55 on dist-test-slave-56vw"
server_key: "c0b57de046599f40f16637849648f4e7"
server_key_iv: "6750e7b40ba7ec097a748b2ed13d3522"
server_key_version: "encryptionkey@0"
I20260928 22:52:55.065181 1617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:52:55.091161 1617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:52:55.093914 1617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:52:55.094115 1617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:52:55.094390 1617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:52:55.094750 1617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:52:55.094789 1617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:55.094839 1617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:52:55.094871 1617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:55.101848 1617 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.130:39919
I20260928 22:52:55.101924 1745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.130:39919 every 8 connection(s)
I20260928 22:52:55.102268 1617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
Sep 28 22:52:55 dist-test-slave-56vw krb5kdc[1376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635975, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.130@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:52:55.111415 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 1617
I20260928 22:52:55.111557 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance
I20260928 22:52:55.111824 16870 external_mini_cluster.cc:1428] Setting key ea9f57ca6c73b56adb4c1daebc62decd
I20260928 22:52:55.112802 1460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:55.103099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:40725 (local address 127.16.121.190:42523)
0928 22:52:55.103158 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:55.103160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:55.103404 (+ 244us) server_negotiation.cc:407] Connection header received
0928 22:52:55.104027 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:55.104029 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:55.104042 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:55.104103 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:52:55.105180 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:55.105737 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:55.106871 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:55.107167 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:55.107270 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:55.109495 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:52:55.109497 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:52:55.109497 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:52:55.109505 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:52:55.111248 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:52:55.111720 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:52:55.111721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:52:55.111722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:52:55.111791 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:52:55.112089 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:52:55.112090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:52:55.112090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:52:55.112216 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:52:55.112297 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:55.112574 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 22:52:55.112670 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 22:52:55.112957 1748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:55.103081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:42523 (local address 127.16.121.130:40725)
0928 22:52:55.103270 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:55.103310 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 22:52:55.103908 (+ 598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:52:55.104189 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:52:55.104193 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:52:55.104830 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:52:55.105043 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:55.105048 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:55.105851 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:55.105853 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:55.106755 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:55.106757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:55.107272 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:55.107273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:55.107415 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:55.107912 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:52:55.107921 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:52:55.109400 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:52:55.111344 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:52:55.111351 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:52:55.111352 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:52:55.111634 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:52:55.111884 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:52:55.111885 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:52:55.111885 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:52:55.112014 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:52:55.112288 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:52:55.112294 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:52:55.112438 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 22:52:55.112551 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 22:52:55.112716 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":87,"threads_started":1}
I20260928 22:52:55.114291 1746 heartbeater.cc:347] Connected to a master server at 127.16.121.190:42523
I20260928 22:52:55.114420 1746 heartbeater.cc:464] Registering TS with master...
I20260928 22:52:55.114622 1746 heartbeater.cc:511] Master 127.16.121.190:42523 requested a full tablet report, sending...
I20260928 22:52:55.115115 1421 ts_manager.cc:194] Registered new tserver with Master: dbfa190872714f7596bda8ecf1f2801d (127.16.121.130:39919)
I20260928 22:52:55.115658 1421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.130@KRBTEST.COM'} at 127.16.121.130:40725
WARNING: no policy specified for kudu/127.16.121.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab.
Entry for principal kudu/127.16.121.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab.
I20260928 22:52:55.141992 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.131:0
--local_ip_for_outbound_sockets=127.16.121.131
--webserver_interface=127.16.121.131
--webserver_port=0
--tserver_master_addrs=127.16.121.190:42523
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab
--principal=kudu/127.16.121.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:39733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:52:55.228986 1751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:52:55.229231 1751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:52:55.229259 1751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:52:55.230499 1751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:52:55.230547 1751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:52:55.230577 1751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.131
I20260928 22:52:55.232231 1751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:39733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab
--principal=kudu/127.16.121.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.121.131
--webserver_port=0
--tserver_master_addrs=127.16.121.190:42523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.131
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:52:55.232441 1751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:52:55.232632 1751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:52:55.237833 1759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:52:55.237833 1757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:52:55.237883 1756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:52:55.238096 1751 server_base.cc:1034] running on GCE node
I20260928 22:52:55.238314 1751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:52:55.238561 1751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:52:55.239732 1751 hybrid_clock.cc:648] HybridClock initialized: now 1790635975239713 us; error 28 us; skew 500 ppm
Sep 28 22:52:55 dist-test-slave-56vw krb5kdc[1376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635975, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:52:55.242599 1751 init.cc:377] Logged in from keytab as kudu/127.16.121.131@KRBTEST.COM (short username kudu)
I20260928 22:52:55.243175 1751 webserver.cc:460] Webserver started at http://127.16.121.131:46251/ using document root <none> and password file <none>
I20260928 22:52:55.243410 1751 fs_manager.cc:359] Metadata directory not provided
I20260928 22:52:55.243479 1751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:52:55.243692 1751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:52:55.244819 1751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-2/data/instance:
uuid: "d957375b1fc34a50adad1dc4e397286d"
format_stamp: "Formatted at 2026-09-28 22:52:55 on dist-test-slave-56vw"
server_key: "37c6ba6c3b577ca88d55573c3af1ddc8"
server_key_iv: "a0d63ba1393f434073328c0123f12b98"
server_key_version: "encryptionkey@0"
I20260928 22:52:55.245208 1751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance:
uuid: "d957375b1fc34a50adad1dc4e397286d"
format_stamp: "Formatted at 2026-09-28 22:52:55 on dist-test-slave-56vw"
server_key: "37c6ba6c3b577ca88d55573c3af1ddc8"
server_key_iv: "a0d63ba1393f434073328c0123f12b98"
server_key_version: "encryptionkey@0"
I20260928 22:52:55.246790 1751 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:52:55.247779 1766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:55.247967 1751 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:52:55.248066 1751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-2/wal
uuid: "d957375b1fc34a50adad1dc4e397286d"
format_stamp: "Formatted at 2026-09-28 22:52:55 on dist-test-slave-56vw"
server_key: "37c6ba6c3b577ca88d55573c3af1ddc8"
server_key_iv: "a0d63ba1393f434073328c0123f12b98"
server_key_version: "encryptionkey@0"
I20260928 22:52:55.248158 1751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:52:55.266745 1751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:52:55.269529 1751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:52:55.269817 1751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:52:55.270120 1751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:52:55.270463 1751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:52:55.270520 1751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:55.270583 1751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:52:55.270625 1751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:55.277176 1751 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.131:41427
I20260928 22:52:55.277238 1879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.131:41427 every 8 connection(s)
I20260928 22:52:55.277603 1751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
Sep 28 22:52:55 dist-test-slave-56vw krb5kdc[1376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635975, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.131@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:52:55.287628 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 1751
I20260928 22:52:55.287741 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance
I20260928 22:52:55.287930 16870 external_mini_cluster.cc:1428] Setting key 1dec9046117d5682a77f7d1610dbf7e2
I20260928 22:52:55.288445 1460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:55.278590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:33243 (local address 127.16.121.190:42523)
0928 22:52:55.278711 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:55.278713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:55.278774 (+ 61us) server_negotiation.cc:407] Connection header received
0928 22:52:55.279421 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:55.279422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:55.279434 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:55.279612 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:52:55.280736 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:55.281310 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:55.282355 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:55.282542 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:55.282594 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:55.285106 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:52:55.285107 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:52:55.285107 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:52:55.285115 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:52:55.286719 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:52:55.287201 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:52:55.287202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:52:55.287203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:52:55.287260 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:52:55.287580 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:52:55.287581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:52:55.287582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:52:55.287759 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:52:55.287830 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:55.288264 (+ 434us) server_negotiation.cc:299] Negotiation successful
0928 22:52:55.288358 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260928 22:52:55.288640 1882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:55.278464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:42523 (local address 127.16.121.131:33243)
0928 22:52:55.278642 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:55.278688 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 22:52:55.279290 (+ 602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:52:55.279687 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:52:55.279691 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:52:55.280359 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:52:55.280576 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:55.280581 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:55.281423 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:55.281423 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:55.282261 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:55.282274 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:55.282621 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:55.282622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:55.282734 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:55.283424 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:52:55.283432 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:52:55.285004 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:52:55.286819 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:52:55.286826 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:52:55.286828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:52:55.287087 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:52:55.287370 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:52:55.287371 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:52:55.287371 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:52:55.287491 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:52:55.287866 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:52:55.287874 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:52:55.288059 (+ 185us) client_negotiation.cc:769] Sending connection context
0928 22:52:55.288194 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 22:52:55.288383 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":97,"threads_started":1}
I20260928 22:52:55.289141 1880 heartbeater.cc:347] Connected to a master server at 127.16.121.190:42523
I20260928 22:52:55.289281 1880 heartbeater.cc:464] Registering TS with master...
I20260928 22:52:55.289474 1880 heartbeater.cc:511] Master 127.16.121.190:42523 requested a full tablet report, sending...
I20260928 22:52:55.289911 1421 ts_manager.cc:194] Registered new tserver with Master: d957375b1fc34a50adad1dc4e397286d (127.16.121.131:41427)
I20260928 22:52:55.290305 1421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.131@KRBTEST.COM'} at 127.16.121.131:33243
I20260928 22:52:55.299932 16870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:52:55.307789 1460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:55.301292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51736 (local address 127.16.121.190:42523)
0928 22:52:55.301336 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:55.301339 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:55.301452 (+ 113us) server_negotiation.cc:407] Connection header received
0928 22:52:55.301503 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:55.301503 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:55.301513 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:55.301554 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:52:55.302130 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:55.302699 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:55.303594 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:55.303854 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:55.303908 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:55.305144 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:52:55.305145 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:52:55.305145 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:52:55.305151 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:52:55.306593 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:52:55.306921 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:52:55.306922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:52:55.306922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:52:55.306982 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:52:55.307279 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:52:55.307279 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:52:55.307280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:52:55.307426 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:52:55.307510 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:55.307648 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 22:52:55.307715 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 22:52:55.310001 1421 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:52:55.311913 1421 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:52:55.319284 1891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:55.314343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.130:39919 (local address 127.0.0.1:52188)
0928 22:52:55.314482 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:55.314526 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 22:52:55.314591 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:52:55.315010 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:52:55.315014 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:52:55.315017 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:52:55.315120 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:55.315122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:55.316494 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:55.316495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:55.317948 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:55.317951 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:55.318938 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:55.318939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:55.319023 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:55.319038 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 22:52:55.319143 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 22:52:55.319202 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":15,"thread_start_us":88,"threads_started":1}
I20260928 22:52:55.319289 1892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:55.314546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52188 (local address 127.16.121.130:39919)
0928 22:52:55.314737 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:55.314742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:55.314755 (+ 13us) server_negotiation.cc:407] Connection header received
0928 22:52:55.314802 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:55.314804 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:55.314822 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:55.314903 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:52:55.315195 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:55.316377 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:55.318044 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:55.318820 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:55.318906 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:55.318977 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:55.319161 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 22:52:55.319206 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":122,"threads_started":1}
I20260928 22:52:55.320240 1894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:55.314782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.131:41427 (local address 127.0.0.1:32994)
0928 22:52:55.314990 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:55.315000 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:52:55.315087 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:52:55.315373 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:52:55.315374 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:52:55.315376 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:52:55.315429 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:55.315431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:55.316651 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:55.316652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:55.317948 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:55.317951 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:55.320000 (+ 2049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:55.320001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:55.320047 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:55.320051 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:52:55.320122 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 22:52:55.320167 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":33,"thread_start_us":66,"threads_started":1}
I20260928 22:52:55.320261 1895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:55.314886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32994 (local address 127.16.121.131:41427)
0928 22:52:55.315104 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:55.315109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:55.315119 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:52:55.315202 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:55.315204 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:55.315219 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:55.315304 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:52:55.315511 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:55.316540 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:55.319017 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:55.319849 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:55.319914 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:55.319972 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:55.320144 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 22:52:55.320184 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":86,"threads_started":1}
I20260928 22:52:55.320156 1680 tablet_service.cc:1467] Processing CreateTablet for tablet e246178397684dca8295af886e91d991 (DEFAULT_TABLE table=test-table [id=22e8a6c58f96475797d0c93bdfa4739c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:52:55.320538 1680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e246178397684dca8295af886e91d991. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:52:55.321017 1893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:55.314579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:33327 (local address 127.0.0.1:37968)
0928 22:52:55.314783 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:55.314795 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:52:55.314854 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:52:55.316717 (+ 1863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:52:55.316718 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:52:55.316720 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:52:55.316766 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:55.316768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:55.317776 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:55.317777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:55.318843 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:55.318846 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:55.320750 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:55.320751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:55.320812 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:55.320816 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:52:55.320878 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 22:52:55.320921 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":95,"threads_started":1}
I20260928 22:52:55.320952 1814 tablet_service.cc:1467] Processing CreateTablet for tablet e246178397684dca8295af886e91d991 (DEFAULT_TABLE table=test-table [id=22e8a6c58f96475797d0c93bdfa4739c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:52:55.321108 1896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:55.314988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37968 (local address 127.16.121.129:33327)
0928 22:52:55.316558 (+ 1570us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:55.316564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:55.316574 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:52:55.316615 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:55.316617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:55.316644 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:55.316717 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:52:55.316823 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:55.317678 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:55.318938 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:55.320653 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:55.320749 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:55.320800 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:55.320913 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 22:52:55.320962 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1531,"thread_start_us":162,"threads_started":1}
I20260928 22:52:55.321309 1814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e246178397684dca8295af886e91d991. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:52:55.323760 1546 tablet_service.cc:1467] Processing CreateTablet for tablet e246178397684dca8295af886e91d991 (DEFAULT_TABLE table=test-table [id=22e8a6c58f96475797d0c93bdfa4739c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:52:55.323935 1897 tablet_bootstrap.cc:492] T e246178397684dca8295af886e91d991 P dbfa190872714f7596bda8ecf1f2801d: Bootstrap starting.
I20260928 22:52:55.324090 1546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e246178397684dca8295af886e91d991. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:52:55.324343 1898 tablet_bootstrap.cc:492] T e246178397684dca8295af886e91d991 P d957375b1fc34a50adad1dc4e397286d: Bootstrap starting.
I20260928 22:52:55.324672 1897 tablet_bootstrap.cc:654] T e246178397684dca8295af886e91d991 P dbfa190872714f7596bda8ecf1f2801d: Neither blocks nor log segments found. Creating new log.
I20260928 22:52:55.324959 1898 tablet_bootstrap.cc:654] T e246178397684dca8295af886e91d991 P d957375b1fc34a50adad1dc4e397286d: Neither blocks nor log segments found. Creating new log.
I20260928 22:52:55.325094 1897 log.cc:824] T e246178397684dca8295af886e91d991 P dbfa190872714f7596bda8ecf1f2801d: Log is configured to *not* fsync() on all Append() calls
I20260928 22:52:55.325543 1898 log.cc:824] T e246178397684dca8295af886e91d991 P d957375b1fc34a50adad1dc4e397286d: Log is configured to *not* fsync() on all Append() calls
I20260928 22:52:55.325781 1897 tablet_bootstrap.cc:492] T e246178397684dca8295af886e91d991 P dbfa190872714f7596bda8ecf1f2801d: No bootstrap required, opened a new log
I20260928 22:52:55.325845 1897 ts_tablet_manager.cc:1397] T e246178397684dca8295af886e91d991 P dbfa190872714f7596bda8ecf1f2801d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:52:55.326342 1898 tablet_bootstrap.cc:492] T e246178397684dca8295af886e91d991 P d957375b1fc34a50adad1dc4e397286d: No bootstrap required, opened a new log
I20260928 22:52:55.326402 1898 ts_tablet_manager.cc:1397] T e246178397684dca8295af886e91d991 P d957375b1fc34a50adad1dc4e397286d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:52:55.327737 1897 raft_consensus.cc:348] T e246178397684dca8295af886e91d991 P dbfa190872714f7596bda8ecf1f2801d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbfa190872714f7596bda8ecf1f2801d" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 39919 } } peers { permanent_uuid: "cc78e47f1b9c4c1ca1ab9f43905781af" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33327 } } peers { permanent_uuid: "d957375b1fc34a50adad1dc4e397286d" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41427 } }
I20260928 22:52:55.327905 1897 raft_consensus.cc:374] T e246178397684dca8295af886e91d991 P dbfa190872714f7596bda8ecf1f2801d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:52:55.327951 1897 raft_consensus.cc:729] T e246178397684dca8295af886e91d991 P dbfa190872714f7596bda8ecf1f2801d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbfa190872714f7596bda8ecf1f2801d, State: Initialized, Role: FOLLOWER
I20260928 22:52:55.328099 1897 consensus_queue.cc:260] T e246178397684dca8295af886e91d991 P dbfa190872714f7596bda8ecf1f2801d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbfa190872714f7596bda8ecf1f2801d" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 39919 } } peers { permanent_uuid: "cc78e47f1b9c4c1ca1ab9f43905781af" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33327 } } peers { permanent_uuid: "d957375b1fc34a50adad1dc4e397286d" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41427 } }
I20260928 22:52:55.328418 1897 ts_tablet_manager.cc:1428] T e246178397684dca8295af886e91d991 P dbfa190872714f7596bda8ecf1f2801d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:52:55.328437 1746 heartbeater.cc:503] Master 127.16.121.190:42523 was elected leader, sending a full tablet report...
I20260928 22:52:55.328408 1898 raft_consensus.cc:348] T e246178397684dca8295af886e91d991 P d957375b1fc34a50adad1dc4e397286d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbfa190872714f7596bda8ecf1f2801d" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 39919 } } peers { permanent_uuid: "cc78e47f1b9c4c1ca1ab9f43905781af" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33327 } } peers { permanent_uuid: "d957375b1fc34a50adad1dc4e397286d" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41427 } }
I20260928 22:52:55.328542 1898 raft_consensus.cc:374] T e246178397684dca8295af886e91d991 P d957375b1fc34a50adad1dc4e397286d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:52:55.328564 1898 raft_consensus.cc:729] T e246178397684dca8295af886e91d991 P d957375b1fc34a50adad1dc4e397286d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d957375b1fc34a50adad1dc4e397286d, State: Initialized, Role: FOLLOWER
I20260928 22:52:55.328642 1898 consensus_queue.cc:260] T e246178397684dca8295af886e91d991 P d957375b1fc34a50adad1dc4e397286d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbfa190872714f7596bda8ecf1f2801d" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 39919 } } peers { permanent_uuid: "cc78e47f1b9c4c1ca1ab9f43905781af" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33327 } } peers { permanent_uuid: "d957375b1fc34a50adad1dc4e397286d" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41427 } }
I20260928 22:52:55.328883 1898 ts_tablet_manager.cc:1428] T e246178397684dca8295af886e91d991 P d957375b1fc34a50adad1dc4e397286d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 22:52:55.328897 1880 heartbeater.cc:503] Master 127.16.121.190:42523 was elected leader, sending a full tablet report...
I20260928 22:52:55.329098 1901 tablet_bootstrap.cc:492] T e246178397684dca8295af886e91d991 P cc78e47f1b9c4c1ca1ab9f43905781af: Bootstrap starting.
I20260928 22:52:55.329772 1901 tablet_bootstrap.cc:654] T e246178397684dca8295af886e91d991 P cc78e47f1b9c4c1ca1ab9f43905781af: Neither blocks nor log segments found. Creating new log.
I20260928 22:52:55.330291 1901 log.cc:824] T e246178397684dca8295af886e91d991 P cc78e47f1b9c4c1ca1ab9f43905781af: Log is configured to *not* fsync() on all Append() calls
I20260928 22:52:55.330946 1901 tablet_bootstrap.cc:492] T e246178397684dca8295af886e91d991 P cc78e47f1b9c4c1ca1ab9f43905781af: No bootstrap required, opened a new log
I20260928 22:52:55.330997 1901 ts_tablet_manager.cc:1397] T e246178397684dca8295af886e91d991 P cc78e47f1b9c4c1ca1ab9f43905781af: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:52:55.332562 1901 raft_consensus.cc:348] T e246178397684dca8295af886e91d991 P cc78e47f1b9c4c1ca1ab9f43905781af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbfa190872714f7596bda8ecf1f2801d" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 39919 } } peers { permanent_uuid: "cc78e47f1b9c4c1ca1ab9f43905781af" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33327 } } peers { permanent_uuid: "d957375b1fc34a50adad1dc4e397286d" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41427 } }
I20260928 22:52:55.332695 1901 raft_consensus.cc:374] T e246178397684dca8295af886e91d991 P cc78e47f1b9c4c1ca1ab9f43905781af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:52:55.332718 1901 raft_consensus.cc:729] T e246178397684dca8295af886e91d991 P cc78e47f1b9c4c1ca1ab9f43905781af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc78e47f1b9c4c1ca1ab9f43905781af, State: Initialized, Role: FOLLOWER
I20260928 22:52:55.332839 1901 consensus_queue.cc:260] T e246178397684dca8295af886e91d991 P cc78e47f1b9c4c1ca1ab9f43905781af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbfa190872714f7596bda8ecf1f2801d" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 39919 } } peers { permanent_uuid: "cc78e47f1b9c4c1ca1ab9f43905781af" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33327 } } peers { permanent_uuid: "d957375b1fc34a50adad1dc4e397286d" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41427 } }
I20260928 22:52:55.333127 1612 heartbeater.cc:503] Master 127.16.121.190:42523 was elected leader, sending a full tablet report...
I20260928 22:52:55.333112 1901 ts_tablet_manager.cc:1428] T e246178397684dca8295af886e91d991 P cc78e47f1b9c4c1ca1ab9f43905781af: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20260928 22:52:55.353084 1747 tablet.cc:2369] T e246178397684dca8295af886e91d991 P dbfa190872714f7596bda8ecf1f2801d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:52:55.433410 1613 tablet.cc:2369] T e246178397684dca8295af886e91d991 P cc78e47f1b9c4c1ca1ab9f43905781af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:52:55.481977 1902 raft_consensus.cc:482] T e246178397684dca8295af886e91d991 P dbfa190872714f7596bda8ecf1f2801d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:52:55.482128 1902 raft_consensus.cc:504] T e246178397684dca8295af886e91d991 P dbfa190872714f7596bda8ecf1f2801d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbfa190872714f7596bda8ecf1f2801d" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 39919 } } peers { permanent_uuid: "cc78e47f1b9c4c1ca1ab9f43905781af" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33327 } } peers { permanent_uuid: "d957375b1fc34a50adad1dc4e397286d" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41427 } }
I20260928 22:52:55.482461 1902 leader_election.cc:288] T e246178397684dca8295af886e91d991 P dbfa190872714f7596bda8ecf1f2801d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc78e47f1b9c4c1ca1ab9f43905781af (127.16.121.129:33327), d957375b1fc34a50adad1dc4e397286d (127.16.121.131:41427)
I20260928 22:52:55.486793 1895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:55.482914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:60707 (local address 127.16.121.131:41427)
0928 22:52:55.482995 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:55.482997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:55.483131 (+ 134us) server_negotiation.cc:407] Connection header received
0928 22:52:55.483190 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:55.483192 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:55.483206 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:55.483279 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:52:55.483416 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:55.484205 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:55.485734 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:55.486376 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:55.486450 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:55.486458 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:55.486667 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 22:52:55.486714 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 22:52:55.486861 1906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:55.482815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.131:41427 (local address 127.16.121.130:60707)
0928 22:52:55.483049 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:55.483078 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 22:52:55.483131 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:52:55.483291 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:52:55.483292 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:52:55.483293 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:52:55.483340 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:55.483341 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:55.484283 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:55.484284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:55.485608 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:55.485611 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:55.486463 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:55.486464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:55.486571 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:55.486577 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 22:52:55.486637 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 22:52:55.486696 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":69,"thread_start_us":146,"threads_started":1}
I20260928 22:52:55.486922 1896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:55.482803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:39877 (local address 127.16.121.129:33327)
0928 22:52:55.482906 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:55.482909 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:55.482921 (+ 12us) server_negotiation.cc:407] Connection header received
0928 22:52:55.482955 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:55.482957 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:55.482973 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:55.483119 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:52:55.483241 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:55.484135 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:55.485734 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:55.486359 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:55.486449 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:55.486458 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:55.486795 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 22:52:55.486841 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 22:52:55.486881 1748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:55.482685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:33327 (local address 127.16.121.130:39877)
0928 22:52:55.482757 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:55.482771 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 22:52:55.482820 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:52:55.483078 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:52:55.483079 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:52:55.483081 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:52:55.483162 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:55.483164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:55.484232 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:55.484234 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:55.485608 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:55.485611 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:55.486463 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:55.486464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:55.486571 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:55.486577 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 22:52:55.486637 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 22:52:55.486696 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":75}
I20260928 22:52:55.487381 1834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e246178397684dca8295af886e91d991" candidate_uuid: "dbfa190872714f7596bda8ecf1f2801d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d957375b1fc34a50adad1dc4e397286d" is_pre_election: true
I20260928 22:52:55.487496 1566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e246178397684dca8295af886e91d991" candidate_uuid: "dbfa190872714f7596bda8ecf1f2801d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc78e47f1b9c4c1ca1ab9f43905781af" is_pre_election: true
I20260928 22:52:55.487571 1834 raft_consensus.cc:2454] T e246178397684dca8295af886e91d991 P d957375b1fc34a50adad1dc4e397286d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbfa190872714f7596bda8ecf1f2801d in term 0.
I20260928 22:52:55.487613 1566 raft_consensus.cc:2454] T e246178397684dca8295af886e91d991 P cc78e47f1b9c4c1ca1ab9f43905781af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbfa190872714f7596bda8ecf1f2801d in term 0.
I20260928 22:52:55.487841 1636 leader_election.cc:302] T e246178397684dca8295af886e91d991 P dbfa190872714f7596bda8ecf1f2801d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d957375b1fc34a50adad1dc4e397286d, dbfa190872714f7596bda8ecf1f2801d; no voters:
I20260928 22:52:55.488013 1902 raft_consensus.cc:2781] T e246178397684dca8295af886e91d991 P dbfa190872714f7596bda8ecf1f2801d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:52:55.488140 1902 raft_consensus.cc:482] T e246178397684dca8295af886e91d991 P dbfa190872714f7596bda8ecf1f2801d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:52:55.488178 1902 raft_consensus.cc:3037] T e246178397684dca8295af886e91d991 P dbfa190872714f7596bda8ecf1f2801d [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:52:55.488915 1902 raft_consensus.cc:504] T e246178397684dca8295af886e91d991 P dbfa190872714f7596bda8ecf1f2801d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbfa190872714f7596bda8ecf1f2801d" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 39919 } } peers { permanent_uuid: "cc78e47f1b9c4c1ca1ab9f43905781af" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33327 } } peers { permanent_uuid: "d957375b1fc34a50adad1dc4e397286d" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41427 } }
I20260928 22:52:55.489076 1902 leader_election.cc:288] T e246178397684dca8295af886e91d991 P dbfa190872714f7596bda8ecf1f2801d [CANDIDATE]: Term 1 election: Requested vote from peers cc78e47f1b9c4c1ca1ab9f43905781af (127.16.121.129:33327), d957375b1fc34a50adad1dc4e397286d (127.16.121.131:41427)
I20260928 22:52:55.489279 1566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e246178397684dca8295af886e91d991" candidate_uuid: "dbfa190872714f7596bda8ecf1f2801d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc78e47f1b9c4c1ca1ab9f43905781af"
I20260928 22:52:55.489380 1566 raft_consensus.cc:3037] T e246178397684dca8295af886e91d991 P cc78e47f1b9c4c1ca1ab9f43905781af [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:52:55.489351 1834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e246178397684dca8295af886e91d991" candidate_uuid: "dbfa190872714f7596bda8ecf1f2801d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d957375b1fc34a50adad1dc4e397286d"
I20260928 22:52:55.489465 1834 raft_consensus.cc:3037] T e246178397684dca8295af886e91d991 P d957375b1fc34a50adad1dc4e397286d [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:52:55.490253 1566 raft_consensus.cc:2454] T e246178397684dca8295af886e91d991 P cc78e47f1b9c4c1ca1ab9f43905781af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbfa190872714f7596bda8ecf1f2801d in term 1.
I20260928 22:52:55.490412 1834 raft_consensus.cc:2454] T e246178397684dca8295af886e91d991 P d957375b1fc34a50adad1dc4e397286d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbfa190872714f7596bda8ecf1f2801d in term 1.
I20260928 22:52:55.490437 1636 leader_election.cc:302] T e246178397684dca8295af886e91d991 P dbfa190872714f7596bda8ecf1f2801d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cc78e47f1b9c4c1ca1ab9f43905781af, dbfa190872714f7596bda8ecf1f2801d; no voters:
I20260928 22:52:55.490556 1902 raft_consensus.cc:2781] T e246178397684dca8295af886e91d991 P dbfa190872714f7596bda8ecf1f2801d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:52:55.490734 1902 raft_consensus.cc:686] T e246178397684dca8295af886e91d991 P dbfa190872714f7596bda8ecf1f2801d [term 1 LEADER]: Becoming Leader. State: Replica: dbfa190872714f7596bda8ecf1f2801d, State: Running, Role: LEADER
I20260928 22:52:55.490840 1902 consensus_queue.cc:237] T e246178397684dca8295af886e91d991 P dbfa190872714f7596bda8ecf1f2801d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbfa190872714f7596bda8ecf1f2801d" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 39919 } } peers { permanent_uuid: "cc78e47f1b9c4c1ca1ab9f43905781af" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33327 } } peers { permanent_uuid: "d957375b1fc34a50adad1dc4e397286d" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41427 } }
I20260928 22:52:55.491762 1421 catalog_manager.cc:5495] T e246178397684dca8295af886e91d991 P dbfa190872714f7596bda8ecf1f2801d reported cstate change: term changed from 0 to 1, leader changed from <none> to dbfa190872714f7596bda8ecf1f2801d (127.16.121.130). New cstate: current_term: 1 leader_uuid: "dbfa190872714f7596bda8ecf1f2801d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbfa190872714f7596bda8ecf1f2801d" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 39919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc78e47f1b9c4c1ca1ab9f43905781af" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d957375b1fc34a50adad1dc4e397286d" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41427 } health_report { overall_health: UNKNOWN } } }
I20260928 22:52:55.501583 1892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:55.498989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52198 (local address 127.16.121.130:39919)
0928 22:52:55.499096 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:55.499098 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:55.499107 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:52:55.499153 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:55.499154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:55.499174 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:55.499250 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:52:55.499372 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:55.499943 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:55.500773 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:55.500980 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:55.501057 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:55.501226 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:52:55.501338 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:52:55.501436 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:55.501455 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 22:52:55.501475 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 22:52:55.503785 1566 raft_consensus.cc:1261] T e246178397684dca8295af886e91d991 P cc78e47f1b9c4c1ca1ab9f43905781af [term 1 FOLLOWER]: Refusing update from remote peer dbfa190872714f7596bda8ecf1f2801d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:52:55.503813 1834 raft_consensus.cc:1261] T e246178397684dca8295af886e91d991 P d957375b1fc34a50adad1dc4e397286d [term 1 FOLLOWER]: Refusing update from remote peer dbfa190872714f7596bda8ecf1f2801d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:52:55.504211 1907 consensus_queue.cc:1035] T e246178397684dca8295af886e91d991 P dbfa190872714f7596bda8ecf1f2801d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc78e47f1b9c4c1ca1ab9f43905781af" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:52:55.504444 1902 consensus_queue.cc:1035] T e246178397684dca8295af886e91d991 P dbfa190872714f7596bda8ecf1f2801d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d957375b1fc34a50adad1dc4e397286d" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:52:55.507341 1911 mvcc.cc:205] Tried to move back new op lower bound from 7334444955659735040 to 7334444955611062272. Current Snapshot: MvccSnapshot[applied={T|T < 7334444955659735040}]
I20260928 22:52:55.509586 1420 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:52:55.509699 1420 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:52:55.510579 1420 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f9ab10c1eb79469bbf5abb53dd2284c0: Sending DeleteTablet for 3 replicas of tablet e246178397684dca8295af886e91d991
I20260928 22:52:55.510942 1680 tablet_service.cc:1514] Processing DeleteTablet for tablet e246178397684dca8295af886e91d991 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:52:55 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:52188
I20260928 22:52:55.511032 1546 tablet_service.cc:1514] Processing DeleteTablet for tablet e246178397684dca8295af886e91d991 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:52:55 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:37968
I20260928 22:52:55.511237 1679 tablet_service.cc:1514] Processing DeleteTablet for tablet e246178397684dca8295af886e91d991 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:52:55 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:52188
I20260928 22:52:55.511364 1814 tablet_service.cc:1514] Processing DeleteTablet for tablet e246178397684dca8295af886e91d991 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:52:55 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:32994
I20260928 22:52:55.511394 1920 tablet_replica.cc:330] T e246178397684dca8295af886e91d991 P cc78e47f1b9c4c1ca1ab9f43905781af: stopping tablet replica
I20260928 22:52:55.511479 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 1483
I20260928 22:52:55.511765 1921 tablet_replica.cc:330] T e246178397684dca8295af886e91d991 P d957375b1fc34a50adad1dc4e397286d: stopping tablet replica
I20260928 22:52:55.511868 1921 raft_consensus.cc:2229] T e246178397684dca8295af886e91d991 P d957375b1fc34a50adad1dc4e397286d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:52:55.511974 1921 raft_consensus.cc:2258] T e246178397684dca8295af886e91d991 P d957375b1fc34a50adad1dc4e397286d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:52:55.512344 1922 tablet_replica.cc:330] T e246178397684dca8295af886e91d991 P dbfa190872714f7596bda8ecf1f2801d: stopping tablet replica
I20260928 22:52:55.512449 1922 raft_consensus.cc:2229] T e246178397684dca8295af886e91d991 P dbfa190872714f7596bda8ecf1f2801d [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:52:55.512470 1921 ts_tablet_manager.cc:1905] T e246178397684dca8295af886e91d991 P d957375b1fc34a50adad1dc4e397286d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:52:55.512554 1922 raft_consensus.cc:2258] T e246178397684dca8295af886e91d991 P dbfa190872714f7596bda8ecf1f2801d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:52:55.512997 1922 ts_tablet_manager.cc:1905] T e246178397684dca8295af886e91d991 P dbfa190872714f7596bda8ecf1f2801d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:52:55.514334 1921 ts_tablet_manager.cc:1918] T e246178397684dca8295af886e91d991 P d957375b1fc34a50adad1dc4e397286d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:52:55.514449 1921 log.cc:1196] T e246178397684dca8295af886e91d991 P d957375b1fc34a50adad1dc4e397286d: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-2/wal/wals/e246178397684dca8295af886e91d991
I20260928 22:52:55.514778 1921 ts_tablet_manager.cc:1939] T e246178397684dca8295af886e91d991 P d957375b1fc34a50adad1dc4e397286d: Deleting consensus metadata
I20260928 22:52:55.514988 1922 ts_tablet_manager.cc:1918] T e246178397684dca8295af886e91d991 P dbfa190872714f7596bda8ecf1f2801d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:52:55.515098 1922 log.cc:1196] T e246178397684dca8295af886e91d991 P dbfa190872714f7596bda8ecf1f2801d: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790635733329745-16870-0/minicluster-data/ts-1/wal/wals/e246178397684dca8295af886e91d991
I20260928 22:52:55.515264 1408 catalog_manager.cc:4841] TS d957375b1fc34a50adad1dc4e397286d (127.16.121.131:41427): tablet e246178397684dca8295af886e91d991 (table test-table [id=22e8a6c58f96475797d0c93bdfa4739c]) successfully deleted
I20260928 22:52:55.515385 1922 ts_tablet_manager.cc:1939] T e246178397684dca8295af886e91d991 P dbfa190872714f7596bda8ecf1f2801d: Deleting consensus metadata
I20260928 22:52:55.516167 1408 catalog_manager.cc:4841] TS dbfa190872714f7596bda8ecf1f2801d (127.16.121.130:39919): tablet e246178397684dca8295af886e91d991 (table test-table [id=22e8a6c58f96475797d0c93bdfa4739c]) successfully deleted
W20260928 22:52:55.516348 1408 catalog_manager.cc:4808] TS dbfa190872714f7596bda8ecf1f2801d (127.16.121.130:39919): delete failed for tablet e246178397684dca8295af886e91d991 because the tablet was not found. No further retry: Not found: Tablet not found: e246178397684dca8295af886e91d991
W20260928 22:52:55.518779 1408 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.121.129:33327 (error 108)
I20260928 22:52:55.519084 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 1617
I20260928 22:52:55.520083 1893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:55.519921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:33327 (local address 127.0.0.1:37980)
0928 22:52:55.520002 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:55.520045 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.129:33327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260928 22:52:55.520231 1408 catalog_manager.cc:4584] TS cc78e47f1b9c4c1ca1ab9f43905781af (127.16.121.129:33327): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e246178397684dca8295af886e91d991: Network error: Client connection negotiation failed: client connection to 127.16.121.129:33327: connect: Connection refused (error 111)
I20260928 22:52:55.524539 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 1751
I20260928 22:52:55.529573 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 1389
2026-09-28T22:52:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2191 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T22:52:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:52:55Z Disabled control of system clock
I20260928 22:52:55.558069 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:39935
--webserver_interface=127.16.121.190
--webserver_port=0
--builtin_ntp_servers=127.16.121.148:37259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:39935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 22:52:55.641582 1927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:52:55.641829 1927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:52:55.641852 1927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:52:55.643128 1927 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:52:55.643177 1927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:52:55.643189 1927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:52:55.643203 1927 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:52:55.643215 1927 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:52:55.644586 1927 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:37259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:39935
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:39935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.121.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:52:55.644816 1927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:52:55.645074 1927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:52:55.650297 1935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:52:55.650333 1932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:52:55.650362 1927 server_base.cc:1034] running on GCE node
W20260928 22:52:55.650300 1933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:52:55.650796 1927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:52:55.651162 1927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:52:55.652359 1927 hybrid_clock.cc:648] HybridClock initialized: now 1790635975652345 us; error 30 us; skew 500 ppm
I20260928 22:52:55.653784 1927 webserver.cc:460] Webserver started at http://127.16.121.190:35189/ using document root <none> and password file <none>
I20260928 22:52:55.654083 1927 fs_manager.cc:359] Metadata directory not provided
I20260928 22:52:55.654131 1927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:52:55.654294 1927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:52:55.655499 1927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "b2de88f296b348b5b7a50be086bfa597"
format_stamp: "Formatted at 2026-09-28 22:52:55 on dist-test-slave-56vw"
server_key: "a8e0229d2be6e975a55d0f2bfda936ca"
server_key_iv: "07aea173e4428139d4f111457f085443"
server_key_version: "encryptionkey@0"
I20260928 22:52:55.655872 1927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "b2de88f296b348b5b7a50be086bfa597"
format_stamp: "Formatted at 2026-09-28 22:52:55 on dist-test-slave-56vw"
server_key: "a8e0229d2be6e975a55d0f2bfda936ca"
server_key_iv: "07aea173e4428139d4f111457f085443"
server_key_version: "encryptionkey@0"
I20260928 22:52:55.657346 1927 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:52:55.658207 1941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:55.658385 1927 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:52:55.658485 1927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "b2de88f296b348b5b7a50be086bfa597"
format_stamp: "Formatted at 2026-09-28 22:52:55 on dist-test-slave-56vw"
server_key: "a8e0229d2be6e975a55d0f2bfda936ca"
server_key_iv: "07aea173e4428139d4f111457f085443"
server_key_version: "encryptionkey@0"
I20260928 22:52:55.658546 1927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:52:55.690339 1927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:52:55.690644 1927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:52:55.690749 1927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:52:55.694903 1927 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.190:39935
I20260928 22:52:55.694977 1993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.190:39935 every 8 connection(s)
I20260928 22:52:55.695369 1927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:52:55.696412 1994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:52:55.698726 1994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2de88f296b348b5b7a50be086bfa597: Bootstrap starting.
I20260928 22:52:55.699309 1994 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2de88f296b348b5b7a50be086bfa597: Neither blocks nor log segments found. Creating new log.
I20260928 22:52:55.699556 1994 log.cc:824] T 00000000000000000000000000000000 P b2de88f296b348b5b7a50be086bfa597: Log is configured to *not* fsync() on all Append() calls
I20260928 22:52:55.700258 1994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2de88f296b348b5b7a50be086bfa597: No bootstrap required, opened a new log
I20260928 22:52:55.701843 1994 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2de88f296b348b5b7a50be086bfa597 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2de88f296b348b5b7a50be086bfa597" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39935 } }
I20260928 22:52:55.702001 1994 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2de88f296b348b5b7a50be086bfa597 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:52:55.702025 1994 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2de88f296b348b5b7a50be086bfa597 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2de88f296b348b5b7a50be086bfa597, State: Initialized, Role: FOLLOWER
I20260928 22:52:55.702191 1994 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2de88f296b348b5b7a50be086bfa597 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2de88f296b348b5b7a50be086bfa597" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39935 } }
I20260928 22:52:55.702279 1994 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2de88f296b348b5b7a50be086bfa597 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:52:55.702303 1994 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2de88f296b348b5b7a50be086bfa597 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:52:55.702332 1994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2de88f296b348b5b7a50be086bfa597 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:52:55.702919 1994 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2de88f296b348b5b7a50be086bfa597 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2de88f296b348b5b7a50be086bfa597" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39935 } }
I20260928 22:52:55.703020 1994 leader_election.cc:302] T 00000000000000000000000000000000 P b2de88f296b348b5b7a50be086bfa597 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2de88f296b348b5b7a50be086bfa597; no voters:
I20260928 22:52:55.703294 1994 leader_election.cc:288] T 00000000000000000000000000000000 P b2de88f296b348b5b7a50be086bfa597 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:52:55.703349 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 1927
I20260928 22:52:55.703392 1997 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2de88f296b348b5b7a50be086bfa597 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:52:55.703465 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:52:55.703586 1997 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2de88f296b348b5b7a50be086bfa597 [term 1 LEADER]: Becoming Leader. State: Replica: b2de88f296b348b5b7a50be086bfa597, State: Running, Role: LEADER
I20260928 22:52:55.703641 1994 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2de88f296b348b5b7a50be086bfa597 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:52:55.703738 16870 external_mini_cluster.cc:1428] Setting key 82ca08b701ccc35f8f772501d7831ce0
I20260928 22:52:55.703722 1997 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2de88f296b348b5b7a50be086bfa597 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2de88f296b348b5b7a50be086bfa597" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39935 } }
I20260928 22:52:55.704244 1998 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2de88f296b348b5b7a50be086bfa597 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2de88f296b348b5b7a50be086bfa597" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2de88f296b348b5b7a50be086bfa597" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39935 } } }
I20260928 22:52:55.704346 1998 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2de88f296b348b5b7a50be086bfa597 [sys.catalog]: This master's current role is: LEADER
I20260928 22:52:55.704632 2007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:52:55.705183 2007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:52:55.705186 1999 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2de88f296b348b5b7a50be086bfa597 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2de88f296b348b5b7a50be086bfa597. Latest consensus state: current_term: 1 leader_uuid: "b2de88f296b348b5b7a50be086bfa597" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2de88f296b348b5b7a50be086bfa597" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 39935 } } }
I20260928 22:52:55.706125 1999 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2de88f296b348b5b7a50be086bfa597 [sys.catalog]: This master's current role is: LEADER
I20260928 22:52:55.707202 2007 catalog_manager.cc:1347] Generated new cluster ID: a4f7bb570cc04f71b8b32aea6fead20d
I20260928 22:52:55.707283 2007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:52:55.710896 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:55.704564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48604 (local address 127.16.121.190:39935)
0928 22:52:55.704954 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:55.704966 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:55.705000 (+ 34us) server_negotiation.cc:407] Connection header received
0928 22:52:55.705159 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:55.705168 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:55.705260 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:55.705502 (+ 242us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:52:55.705998 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:55.707402 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:55.708630 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:55.709278 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:55.709411 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:55.710164 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:52:55.710168 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:52:55.710168 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:52:55.710389 (+ 221us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:52:55.710433 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:52:55.710448 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:52:55.710520 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:55.710592 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 22:52:55.710675 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":89,"threads_started":1}
I20260928 22:52:55.717514 2007 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:52:55.718098 2007 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:52:55.723804 2007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2de88f296b348b5b7a50be086bfa597: Generated new TSK 0
I20260928 22:52:55.724027 2007 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:52:55.727447 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.129:0
--local_ip_for_outbound_sockets=127.16.121.129
--webserver_interface=127.16.121.129
--webserver_port=0
--tserver_master_addrs=127.16.121.190:39935
--builtin_ntp_servers=127.16.121.148:37259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 22:52:55.815798 2018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:52:55.816069 2018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:52:55.816110 2018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:52:55.817325 2018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:52:55.817391 2018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:52:55.817436 2018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.129
I20260928 22:52:55.818847 2018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:37259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.121.129
--webserver_port=0
--tserver_master_addrs=127.16.121.190:39935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.129
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:52:55.819140 2018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:52:55.819458 2018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:52:55.824705 2024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:52:55.824769 2023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:52:55.824864 2026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:52:55.825227 2018 server_base.cc:1034] running on GCE node
I20260928 22:52:55.825497 2018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:52:55.825799 2018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:52:55.826974 2018 hybrid_clock.cc:648] HybridClock initialized: now 1790635975826960 us; error 32 us; skew 500 ppm
I20260928 22:52:55.828511 2018 webserver.cc:460] Webserver started at http://127.16.121.129:42135/ using document root <none> and password file <none>
I20260928 22:52:55.828812 2018 fs_manager.cc:359] Metadata directory not provided
I20260928 22:52:55.828891 2018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:52:55.829049 2018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:52:55.830175 2018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-0/data/instance:
uuid: "aeba066be41546f5b540457cf2bddcec"
format_stamp: "Formatted at 2026-09-28 22:52:55 on dist-test-slave-56vw"
server_key: "75bcb2413960a194034303de7956b560"
server_key_iv: "fbe9023dc53c621289e67db2db4588a7"
server_key_version: "encryptionkey@0"
I20260928 22:52:55.830566 2018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance:
uuid: "aeba066be41546f5b540457cf2bddcec"
format_stamp: "Formatted at 2026-09-28 22:52:55 on dist-test-slave-56vw"
server_key: "75bcb2413960a194034303de7956b560"
server_key_iv: "fbe9023dc53c621289e67db2db4588a7"
server_key_version: "encryptionkey@0"
I20260928 22:52:55.832074 2018 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:52:55.832921 2032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:55.833097 2018 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:52:55.833199 2018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-0/wal
uuid: "aeba066be41546f5b540457cf2bddcec"
format_stamp: "Formatted at 2026-09-28 22:52:55 on dist-test-slave-56vw"
server_key: "75bcb2413960a194034303de7956b560"
server_key_iv: "fbe9023dc53c621289e67db2db4588a7"
server_key_version: "encryptionkey@0"
I20260928 22:52:55.833312 2018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:52:55.842536 2018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:52:55.842846 2018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:52:55.842993 2018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:52:55.843247 2018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:52:55.843567 2018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:52:55.843619 2018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:55.843662 2018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:52:55.843691 2018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:55.849571 2018 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.129:33291
I20260928 22:52:55.849617 2145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.129:33291 every 8 connection(s)
I20260928 22:52:55.849969 2018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
I20260928 22:52:55.854341 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 2018
I20260928 22:52:55.854453 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance
I20260928 22:52:55.854661 16870 external_mini_cluster.cc:1428] Setting key 5f96986b134a8bbe296929f4537c9f4a
I20260928 22:52:55.855935 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:55.850791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:40089 (local address 127.16.121.190:39935)
0928 22:52:55.850839 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:55.850841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:55.851151 (+ 310us) server_negotiation.cc:407] Connection header received
0928 22:52:55.851663 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:55.851664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:55.851680 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:55.851740 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:52:55.852402 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:55.852909 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:55.854030 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:55.854322 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:55.854445 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:55.855503 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:52:55.855504 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:52:55.855504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:52:55.855510 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:52:55.855531 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:52:55.855537 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:52:55.855604 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:55.855839 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 22:52:55.855873 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 22:52:55.855998 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.130:0
--local_ip_for_outbound_sockets=127.16.121.130
--webserver_interface=127.16.121.130
--webserver_port=0
--tserver_master_addrs=127.16.121.190:39935
--builtin_ntp_servers=127.16.121.148:37259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 22:52:55.856261 2148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:55.850783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:39935 (local address 127.16.121.129:40089)
0928 22:52:55.850963 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:55.850999 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 22:52:55.851572 (+ 573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:52:55.851795 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:52:55.851798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:52:55.851806 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:52:55.852319 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:55.852323 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:55.853014 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:55.853015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:55.853942 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:55.853945 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:55.854441 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:55.854443 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:55.854607 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:55.855311 (+ 704us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:52:55.855311 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:52:55.855347 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:52:55.855402 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:52:55.855689 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:52:55.855696 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:52:55.855701 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:52:55.855796 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 22:52:55.855965 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":79,"threads_started":1}
I20260928 22:52:55.857736 2146 heartbeater.cc:347] Connected to a master server at 127.16.121.190:39935
I20260928 22:52:55.857833 2146 heartbeater.cc:464] Registering TS with master...
I20260928 22:52:55.858114 2146 heartbeater.cc:511] Master 127.16.121.190:39935 requested a full tablet report, sending...
I20260928 22:52:55.858745 1958 ts_manager.cc:194] Registered new tserver with Master: aeba066be41546f5b540457cf2bddcec (127.16.121.129:33291)
I20260928 22:52:55.859501 1958 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.121.129:40089
W20260928 22:52:55.940937 2149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:52:55.941191 2149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:52:55.941216 2149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:52:55.942781 2149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:52:55.942838 2149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:52:55.942865 2149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.130
I20260928 22:52:55.944336 2149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:37259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.121.130
--webserver_port=0
--tserver_master_addrs=127.16.121.190:39935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.130
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:52:55.944617 2149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:52:55.944937 2149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:52:55.950317 2154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:52:55.950356 2157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:52:55.950366 2149 server_base.cc:1034] running on GCE node
W20260928 22:52:55.950311 2155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:52:55.950786 2149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:52:55.951180 2149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:52:55.952349 2149 hybrid_clock.cc:648] HybridClock initialized: now 1790635975952340 us; error 30 us; skew 500 ppm
I20260928 22:52:55.953709 2149 webserver.cc:460] Webserver started at http://127.16.121.130:41395/ using document root <none> and password file <none>
I20260928 22:52:55.953965 2149 fs_manager.cc:359] Metadata directory not provided
I20260928 22:52:55.954016 2149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:52:55.954167 2149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:52:55.955492 2149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-1/data/instance:
uuid: "d384f23e70ae46bc869ee4c7d3a08fae"
format_stamp: "Formatted at 2026-09-28 22:52:55 on dist-test-slave-56vw"
server_key: "f2f4be50643a282df3ae8287a2ec6f10"
server_key_iv: "e4fb1988ccbeb602141997589a17cce8"
server_key_version: "encryptionkey@0"
I20260928 22:52:55.955886 2149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance:
uuid: "d384f23e70ae46bc869ee4c7d3a08fae"
format_stamp: "Formatted at 2026-09-28 22:52:55 on dist-test-slave-56vw"
server_key: "f2f4be50643a282df3ae8287a2ec6f10"
server_key_iv: "e4fb1988ccbeb602141997589a17cce8"
server_key_version: "encryptionkey@0"
I20260928 22:52:55.957278 2149 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:52:55.958124 2163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:55.958389 2149 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:52:55.958555 2149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-1/wal
uuid: "d384f23e70ae46bc869ee4c7d3a08fae"
format_stamp: "Formatted at 2026-09-28 22:52:55 on dist-test-slave-56vw"
server_key: "f2f4be50643a282df3ae8287a2ec6f10"
server_key_iv: "e4fb1988ccbeb602141997589a17cce8"
server_key_version: "encryptionkey@0"
I20260928 22:52:55.958704 2149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:52:55.969805 2149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:52:55.970082 2149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:52:55.970186 2149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:52:55.970374 2149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:52:55.970733 2149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:52:55.970769 2149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:55.970793 2149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:52:55.970842 2149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:55.976692 2149 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.130:43421
I20260928 22:52:55.976804 2276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.130:43421 every 8 connection(s)
I20260928 22:52:55.977141 2149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
I20260928 22:52:55.981753 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 2149
I20260928 22:52:55.981886 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance
I20260928 22:52:55.982136 16870 external_mini_cluster.cc:1428] Setting key d8de947a4e100207d984a8ad88c6453a
I20260928 22:52:55.982612 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:55.978026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:33855 (local address 127.16.121.190:39935)
0928 22:52:55.978112 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:55.978114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:55.978294 (+ 180us) server_negotiation.cc:407] Connection header received
0928 22:52:55.978775 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:55.978776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:55.978786 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:55.978851 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:52:55.979613 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:55.980122 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:55.981093 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:55.981282 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:55.981341 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:55.982299 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:52:55.982300 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:52:55.982300 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:52:55.982305 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:52:55.982321 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:52:55.982325 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:52:55.982373 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:55.982502 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 22:52:55.982532 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 22:52:55.982810 2279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:55.977948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:39935 (local address 127.16.121.130:33855)
0928 22:52:55.978149 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:55.978192 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 22:52:55.978689 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:52:55.978916 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:52:55.978920 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:52:55.978928 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:52:55.979521 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:55.979527 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:55.980207 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:55.980208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:55.981004 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:55.981006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:55.981363 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:55.981364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:55.981473 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:55.982121 (+ 648us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:52:55.982121 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:52:55.982161 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:52:55.982193 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:52:55.982395 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:52:55.982402 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:52:55.982405 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:52:55.982481 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 22:52:55.982582 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260928 22:52:55.983253 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.131:0
--local_ip_for_outbound_sockets=127.16.121.131
--webserver_interface=127.16.121.131
--webserver_port=0
--tserver_master_addrs=127.16.121.190:39935
--builtin_ntp_servers=127.16.121.148:37259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 22:52:55.983357 2277 heartbeater.cc:347] Connected to a master server at 127.16.121.190:39935
I20260928 22:52:55.983465 2277 heartbeater.cc:464] Registering TS with master...
I20260928 22:52:55.983731 2277 heartbeater.cc:511] Master 127.16.121.190:39935 requested a full tablet report, sending...
I20260928 22:52:55.984118 1958 ts_manager.cc:194] Registered new tserver with Master: d384f23e70ae46bc869ee4c7d3a08fae (127.16.121.130:43421)
I20260928 22:52:55.984633 1958 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.121.130:33855
W20260928 22:52:56.068770 2280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:52:56.068984 2280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:52:56.069010 2280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:52:56.070235 2280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:52:56.070278 2280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:52:56.070304 2280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.131
I20260928 22:52:56.071662 2280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:37259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.121.131
--webserver_port=0
--tserver_master_addrs=127.16.121.190:39935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.131
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:52:56.071906 2280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:52:56.072201 2280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:52:56.077195 2286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:52:56.077275 2288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:52:56.077231 2285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:52:56.077389 2280 server_base.cc:1034] running on GCE node
I20260928 22:52:56.077589 2280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:52:56.077821 2280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:52:56.078976 2280 hybrid_clock.cc:648] HybridClock initialized: now 1790635976078970 us; error 36 us; skew 500 ppm
I20260928 22:52:56.080178 2280 webserver.cc:460] Webserver started at http://127.16.121.131:37041/ using document root <none> and password file <none>
I20260928 22:52:56.080452 2280 fs_manager.cc:359] Metadata directory not provided
I20260928 22:52:56.080518 2280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:52:56.080650 2280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:52:56.081687 2280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-2/data/instance:
uuid: "d965e6a28e4f46deb1de3fb7150a3607"
format_stamp: "Formatted at 2026-09-28 22:52:56 on dist-test-slave-56vw"
server_key: "feee0482cef6a33924dad4c44640c625"
server_key_iv: "c4439dedbe26ed52bb3dc084201ad842"
server_key_version: "encryptionkey@0"
I20260928 22:52:56.082087 2280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance:
uuid: "d965e6a28e4f46deb1de3fb7150a3607"
format_stamp: "Formatted at 2026-09-28 22:52:56 on dist-test-slave-56vw"
server_key: "feee0482cef6a33924dad4c44640c625"
server_key_iv: "c4439dedbe26ed52bb3dc084201ad842"
server_key_version: "encryptionkey@0"
I20260928 22:52:56.083410 2280 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:52:56.084118 2294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:56.084296 2280 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:52:56.084388 2280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-2/wal
uuid: "d965e6a28e4f46deb1de3fb7150a3607"
format_stamp: "Formatted at 2026-09-28 22:52:56 on dist-test-slave-56vw"
server_key: "feee0482cef6a33924dad4c44640c625"
server_key_iv: "c4439dedbe26ed52bb3dc084201ad842"
server_key_version: "encryptionkey@0"
I20260928 22:52:56.084470 2280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:52:56.104517 2280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:52:56.104919 2280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:52:56.105041 2280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:52:56.105266 2280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:52:56.105669 2280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:52:56.105706 2280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:56.105729 2280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:52:56.105742 2280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:56.112498 2280 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.131:41285
I20260928 22:52:56.112632 2407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.131:41285 every 8 connection(s)
I20260928 22:52:56.112943 2280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
I20260928 22:52:56.118993 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 2280
I20260928 22:52:56.119117 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:56.113852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:52667 (local address 127.16.121.190:39935)
0928 22:52:56.114071 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:56.114074 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:56.114276 (+ 202us) server_negotiation.cc:407] Connection header received
0928 22:52:56.114845 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:56.114846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:56.114858 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:56.114912 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:52:56.115736 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:56.116254 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:56.117317 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:56.117517 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:56.117606 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:56.118757 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:52:56.118759 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:52:56.118759 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:52:56.118773 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:52:56.118787 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:52:56.118791 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:52:56.118854 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:56.118953 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 22:52:56.119000 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184}
I20260928 22:52:56.119168 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance
I20260928 22:52:56.119334 2410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:56.113759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:39935 (local address 127.16.121.131:52667)
0928 22:52:56.114135 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:56.114182 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 22:52:56.114730 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:52:56.115013 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:52:56.115017 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:52:56.115026 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:52:56.115614 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:56.115620 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:56.116357 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:56.116359 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:56.117211 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:56.117214 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:56.117621 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:56.117622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:56.117785 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:56.118570 (+ 785us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:52:56.118570 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:52:56.118607 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:52:56.118635 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:52:56.118855 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:52:56.118862 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:52:56.118865 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:52:56.118957 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 22:52:56.119111 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":95,"threads_started":1}
I20260928 22:52:56.119419 16870 external_mini_cluster.cc:1428] Setting key d4c42ea8e4dc89130ef0feee6c6aec0f
I20260928 22:52:56.120276 2408 heartbeater.cc:347] Connected to a master server at 127.16.121.190:39935
I20260928 22:52:56.120432 2408 heartbeater.cc:464] Registering TS with master...
I20260928 22:52:56.120675 2408 heartbeater.cc:511] Master 127.16.121.190:39935 requested a full tablet report, sending...
I20260928 22:52:56.121311 1957 ts_manager.cc:194] Registered new tserver with Master: d965e6a28e4f46deb1de3fb7150a3607 (127.16.121.131:41285)
I20260928 22:52:56.121797 1957 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.121.131:52667
I20260928 22:52:56.132570 16870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:52:56.138413 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:56.134236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48620 (local address 127.16.121.190:39935)
0928 22:52:56.134371 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:56.134374 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:56.134486 (+ 112us) server_negotiation.cc:407] Connection header received
0928 22:52:56.134747 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:56.134749 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:56.134766 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:56.134844 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:52:56.135069 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:56.135783 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:56.136827 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:56.137049 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:56.137165 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:56.138034 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:52:56.138036 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:52:56.138036 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:52:56.138042 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:52:56.138062 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:52:56.138067 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:52:56.138193 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:56.138269 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 22:52:56.138302 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 22:52:56.141321 1957 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48620:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:52:56.143306 1957 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:52:56.151757 2421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:56.147690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.131:41285 (local address 127.0.0.1:40560)
0928 22:52:56.147830 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:56.147840 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:52:56.148010 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:52:56.148265 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:52:56.148266 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:52:56.148268 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:52:56.148322 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:56.148324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:56.149457 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:56.149457 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:56.150436 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:56.150438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:56.151430 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:56.151431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:56.151514 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:56.151531 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 22:52:56.151609 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 22:52:56.151664 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":99,"threads_started":1}
I20260928 22:52:56.151765 2419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:56.147513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40560 (local address 127.16.121.131:41285)
0928 22:52:56.147702 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:56.147707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:56.147918 (+ 211us) server_negotiation.cc:407] Connection header received
0928 22:52:56.148133 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:56.148136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:56.148191 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:56.148268 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:52:56.148383 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:56.149369 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:56.150539 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:56.151345 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:56.151429 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:56.151531 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:56.151620 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 22:52:56.151662 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":81,"threads_started":1}
I20260928 22:52:56.152591 2418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:56.147174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54510 (local address 127.16.121.129:33291)
0928 22:52:56.147366 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:56.147372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:56.147386 (+ 14us) server_negotiation.cc:407] Connection header received
0928 22:52:56.147509 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:56.147512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:56.147537 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:56.147636 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:52:56.147832 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:56.148797 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:56.150786 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:56.151551 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:56.151624 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:56.151697 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:56.152442 (+ 745us) server_negotiation.cc:299] Negotiation successful
0928 22:52:56.152500 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":99,"threads_started":1}
I20260928 22:52:56.152618 2342 tablet_service.cc:1467] Processing CreateTablet for tablet 2b6d61d84eb94287b268dbbd6c158c9e (DEFAULT_TABLE table=test-table [id=6c04791b548c401093098b4660d8c46c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:52:56.152827 2416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:56.146962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:33291 (local address 127.0.0.1:54510)
0928 22:52:56.147236 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:56.147290 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 22:52:56.147378 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:52:56.147649 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:52:56.147653 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:52:56.147655 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:52:56.147738 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:56.147741 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:56.148901 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:56.148902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:56.150658 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:56.150662 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:56.152128 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:56.152129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:56.152329 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:56.152335 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 22:52:56.152694 (+ 359us) client_negotiation.cc:240] Negotiation successful
0928 22:52:56.152754 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":40,"thread_start_us":215,"threads_started":1}
I20260928 22:52:56.153065 2342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b6d61d84eb94287b268dbbd6c158c9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:52:56.153493 2080 tablet_service.cc:1467] Processing CreateTablet for tablet 2b6d61d84eb94287b268dbbd6c158c9e (DEFAULT_TABLE table=test-table [id=6c04791b548c401093098b4660d8c46c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:52:56.153824 2080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b6d61d84eb94287b268dbbd6c158c9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:52:56.154031 2417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:56.147143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.130:43421 (local address 127.0.0.1:33442)
0928 22:52:56.147322 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:56.147333 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:52:56.147388 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:52:56.149295 (+ 1907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:52:56.149295 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:52:56.149298 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:52:56.149343 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:56.149345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:56.151307 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:56.151308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:56.152406 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:56.152409 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:56.153800 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:56.153801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:56.153868 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:56.153872 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:52:56.153903 (+ 31us) client_negotiation.cc:240] Negotiation successful
0928 22:52:56.153948 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":110,"threads_started":1}
I20260928 22:52:56.154626 2420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:56.147535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33442 (local address 127.16.121.130:43421)
0928 22:52:56.148997 (+ 1462us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:56.149002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:56.149012 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:52:56.149051 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:56.149053 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:56.149069 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:56.149280 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:52:56.149424 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:56.150957 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:56.152519 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:56.153683 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:56.154250 (+ 567us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:56.154352 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:56.154379 (+ 27us) server_negotiation.cc:299] Negotiation successful
0928 22:52:56.154420 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1410,"thread_start_us":79,"threads_started":1}
I20260928 22:52:56.155773 2211 tablet_service.cc:1467] Processing CreateTablet for tablet 2b6d61d84eb94287b268dbbd6c158c9e (DEFAULT_TABLE table=test-table [id=6c04791b548c401093098b4660d8c46c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:52:56.156127 2211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b6d61d84eb94287b268dbbd6c158c9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:52:56.159264 2424 tablet_bootstrap.cc:492] T 2b6d61d84eb94287b268dbbd6c158c9e P d384f23e70ae46bc869ee4c7d3a08fae: Bootstrap starting.
I20260928 22:52:56.159929 2424 tablet_bootstrap.cc:654] T 2b6d61d84eb94287b268dbbd6c158c9e P d384f23e70ae46bc869ee4c7d3a08fae: Neither blocks nor log segments found. Creating new log.
I20260928 22:52:56.160490 2424 log.cc:824] T 2b6d61d84eb94287b268dbbd6c158c9e P d384f23e70ae46bc869ee4c7d3a08fae: Log is configured to *not* fsync() on all Append() calls
I20260928 22:52:56.161409 2424 tablet_bootstrap.cc:492] T 2b6d61d84eb94287b268dbbd6c158c9e P d384f23e70ae46bc869ee4c7d3a08fae: No bootstrap required, opened a new log
I20260928 22:52:56.161469 2424 ts_tablet_manager.cc:1397] T 2b6d61d84eb94287b268dbbd6c158c9e P d384f23e70ae46bc869ee4c7d3a08fae: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:52:56.163542 2423 tablet_bootstrap.cc:492] T 2b6d61d84eb94287b268dbbd6c158c9e P aeba066be41546f5b540457cf2bddcec: Bootstrap starting.
I20260928 22:52:56.164597 2423 tablet_bootstrap.cc:654] T 2b6d61d84eb94287b268dbbd6c158c9e P aeba066be41546f5b540457cf2bddcec: Neither blocks nor log segments found. Creating new log.
I20260928 22:52:56.164513 2424 raft_consensus.cc:348] T 2b6d61d84eb94287b268dbbd6c158c9e P d384f23e70ae46bc869ee4c7d3a08fae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeba066be41546f5b540457cf2bddcec" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33291 } } peers { permanent_uuid: "d965e6a28e4f46deb1de3fb7150a3607" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41285 } } peers { permanent_uuid: "d384f23e70ae46bc869ee4c7d3a08fae" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 43421 } }
I20260928 22:52:56.164961 2424 raft_consensus.cc:374] T 2b6d61d84eb94287b268dbbd6c158c9e P d384f23e70ae46bc869ee4c7d3a08fae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:52:56.164974 2423 log.cc:824] T 2b6d61d84eb94287b268dbbd6c158c9e P aeba066be41546f5b540457cf2bddcec: Log is configured to *not* fsync() on all Append() calls
I20260928 22:52:56.165037 2424 raft_consensus.cc:729] T 2b6d61d84eb94287b268dbbd6c158c9e P d384f23e70ae46bc869ee4c7d3a08fae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d384f23e70ae46bc869ee4c7d3a08fae, State: Initialized, Role: FOLLOWER
I20260928 22:52:56.165266 2424 consensus_queue.cc:260] T 2b6d61d84eb94287b268dbbd6c158c9e P d384f23e70ae46bc869ee4c7d3a08fae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeba066be41546f5b540457cf2bddcec" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33291 } } peers { permanent_uuid: "d965e6a28e4f46deb1de3fb7150a3607" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41285 } } peers { permanent_uuid: "d384f23e70ae46bc869ee4c7d3a08fae" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 43421 } }
I20260928 22:52:56.165479 2422 tablet_bootstrap.cc:492] T 2b6d61d84eb94287b268dbbd6c158c9e P d965e6a28e4f46deb1de3fb7150a3607: Bootstrap starting.
I20260928 22:52:56.165604 2424 ts_tablet_manager.cc:1428] T 2b6d61d84eb94287b268dbbd6c158c9e P d384f23e70ae46bc869ee4c7d3a08fae: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 22:52:56.165915 2277 heartbeater.cc:503] Master 127.16.121.190:39935 was elected leader, sending a full tablet report...
I20260928 22:52:56.166091 2423 tablet_bootstrap.cc:492] T 2b6d61d84eb94287b268dbbd6c158c9e P aeba066be41546f5b540457cf2bddcec: No bootstrap required, opened a new log
I20260928 22:52:56.166152 2423 ts_tablet_manager.cc:1397] T 2b6d61d84eb94287b268dbbd6c158c9e P aeba066be41546f5b540457cf2bddcec: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 22:52:56.166544 2422 tablet_bootstrap.cc:654] T 2b6d61d84eb94287b268dbbd6c158c9e P d965e6a28e4f46deb1de3fb7150a3607: Neither blocks nor log segments found. Creating new log.
I20260928 22:52:56.166893 2422 log.cc:824] T 2b6d61d84eb94287b268dbbd6c158c9e P d965e6a28e4f46deb1de3fb7150a3607: Log is configured to *not* fsync() on all Append() calls
I20260928 22:52:56.167824 2422 tablet_bootstrap.cc:492] T 2b6d61d84eb94287b268dbbd6c158c9e P d965e6a28e4f46deb1de3fb7150a3607: No bootstrap required, opened a new log
I20260928 22:52:56.167917 2422 ts_tablet_manager.cc:1397] T 2b6d61d84eb94287b268dbbd6c158c9e P d965e6a28e4f46deb1de3fb7150a3607: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 22:52:56.169317 2423 raft_consensus.cc:348] T 2b6d61d84eb94287b268dbbd6c158c9e P aeba066be41546f5b540457cf2bddcec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeba066be41546f5b540457cf2bddcec" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33291 } } peers { permanent_uuid: "d965e6a28e4f46deb1de3fb7150a3607" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41285 } } peers { permanent_uuid: "d384f23e70ae46bc869ee4c7d3a08fae" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 43421 } }
I20260928 22:52:56.169526 2423 raft_consensus.cc:374] T 2b6d61d84eb94287b268dbbd6c158c9e P aeba066be41546f5b540457cf2bddcec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:52:56.169581 2423 raft_consensus.cc:729] T 2b6d61d84eb94287b268dbbd6c158c9e P aeba066be41546f5b540457cf2bddcec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeba066be41546f5b540457cf2bddcec, State: Initialized, Role: FOLLOWER
I20260928 22:52:56.169725 2423 consensus_queue.cc:260] T 2b6d61d84eb94287b268dbbd6c158c9e P aeba066be41546f5b540457cf2bddcec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeba066be41546f5b540457cf2bddcec" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33291 } } peers { permanent_uuid: "d965e6a28e4f46deb1de3fb7150a3607" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41285 } } peers { permanent_uuid: "d384f23e70ae46bc869ee4c7d3a08fae" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 43421 } }
I20260928 22:52:56.170038 2423 ts_tablet_manager.cc:1428] T 2b6d61d84eb94287b268dbbd6c158c9e P aeba066be41546f5b540457cf2bddcec: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 22:52:56.170115 2146 heartbeater.cc:503] Master 127.16.121.190:39935 was elected leader, sending a full tablet report...
I20260928 22:52:56.170857 2422 raft_consensus.cc:348] T 2b6d61d84eb94287b268dbbd6c158c9e P d965e6a28e4f46deb1de3fb7150a3607 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeba066be41546f5b540457cf2bddcec" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33291 } } peers { permanent_uuid: "d965e6a28e4f46deb1de3fb7150a3607" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41285 } } peers { permanent_uuid: "d384f23e70ae46bc869ee4c7d3a08fae" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 43421 } }
I20260928 22:52:56.171007 2422 raft_consensus.cc:374] T 2b6d61d84eb94287b268dbbd6c158c9e P d965e6a28e4f46deb1de3fb7150a3607 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:52:56.171031 2422 raft_consensus.cc:729] T 2b6d61d84eb94287b268dbbd6c158c9e P d965e6a28e4f46deb1de3fb7150a3607 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d965e6a28e4f46deb1de3fb7150a3607, State: Initialized, Role: FOLLOWER
I20260928 22:52:56.171262 2422 consensus_queue.cc:260] T 2b6d61d84eb94287b268dbbd6c158c9e P d965e6a28e4f46deb1de3fb7150a3607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeba066be41546f5b540457cf2bddcec" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33291 } } peers { permanent_uuid: "d965e6a28e4f46deb1de3fb7150a3607" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41285 } } peers { permanent_uuid: "d384f23e70ae46bc869ee4c7d3a08fae" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 43421 } }
I20260928 22:52:56.171748 2422 ts_tablet_manager.cc:1428] T 2b6d61d84eb94287b268dbbd6c158c9e P d965e6a28e4f46deb1de3fb7150a3607: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260928 22:52:56.171830 2408 heartbeater.cc:503] Master 127.16.121.190:39935 was elected leader, sending a full tablet report...
W20260928 22:52:56.227752 2278 tablet.cc:2369] T 2b6d61d84eb94287b268dbbd6c158c9e P d384f23e70ae46bc869ee4c7d3a08fae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:52:56.346263 2430 raft_consensus.cc:482] T 2b6d61d84eb94287b268dbbd6c158c9e P d965e6a28e4f46deb1de3fb7150a3607 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:52:56.346401 2430 raft_consensus.cc:504] T 2b6d61d84eb94287b268dbbd6c158c9e P d965e6a28e4f46deb1de3fb7150a3607 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeba066be41546f5b540457cf2bddcec" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33291 } } peers { permanent_uuid: "d965e6a28e4f46deb1de3fb7150a3607" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41285 } } peers { permanent_uuid: "d384f23e70ae46bc869ee4c7d3a08fae" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 43421 } }
I20260928 22:52:56.347031 2430 leader_election.cc:288] T 2b6d61d84eb94287b268dbbd6c158c9e P d965e6a28e4f46deb1de3fb7150a3607 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aeba066be41546f5b540457cf2bddcec (127.16.121.129:33291), d384f23e70ae46bc869ee4c7d3a08fae (127.16.121.130:43421)
I20260928 22:52:56.349967 2420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:56.347404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:44031 (local address 127.16.121.130:43421)
0928 22:52:56.347486 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:56.347489 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:56.347497 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:52:56.347532 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:56.347533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:56.347543 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:56.347611 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:52:56.347777 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:56.348261 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:56.349217 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:56.349590 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:56.349686 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:56.349695 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:56.349846 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 22:52:56.349899 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 22:52:56.349967 2410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:56.347274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.130:43421 (local address 127.16.121.131:44031)
0928 22:52:56.347333 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:56.347346 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:52:56.347399 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:52:56.347616 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:52:56.347617 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:52:56.347619 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:52:56.347693 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:56.347695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:56.348347 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:56.348348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:56.349118 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:56.349120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:56.349687 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:56.349688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:56.349761 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:56.349765 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:52:56.349839 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 22:52:56.349889 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 22:52:56.350180 2418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:56.347360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:35991 (local address 127.16.121.129:33291)
0928 22:52:56.347414 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:56.347416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:56.347698 (+ 282us) server_negotiation.cc:407] Connection header received
0928 22:52:56.347731 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:56.347732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:56.347745 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:56.347820 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:52:56.347983 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:56.348472 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:56.349408 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:56.349778 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:56.349867 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:56.349874 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:56.350018 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 22:52:56.350068 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 22:52:56.350360 2431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:56.347278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:33291 (local address 127.16.121.131:35991)
0928 22:52:56.347581 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:56.347589 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 22:52:56.347671 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:52:56.347825 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:52:56.347826 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:52:56.347827 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:52:56.347901 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:56.347902 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:56.348560 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:56.348561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:56.349312 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:56.349314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:56.349867 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:56.349868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:56.349942 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:56.349945 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:52:56.350013 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 22:52:56.350201 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":109,"thread_start_us":129,"threads_started":1}
I20260928 22:52:56.350365 2231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b6d61d84eb94287b268dbbd6c158c9e" candidate_uuid: "d965e6a28e4f46deb1de3fb7150a3607" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d384f23e70ae46bc869ee4c7d3a08fae" is_pre_election: true
I20260928 22:52:56.350575 2231 raft_consensus.cc:2454] T 2b6d61d84eb94287b268dbbd6c158c9e P d384f23e70ae46bc869ee4c7d3a08fae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d965e6a28e4f46deb1de3fb7150a3607 in term 0.
W20260928 22:52:56.350754 2147 tablet.cc:2369] T 2b6d61d84eb94287b268dbbd6c158c9e P aeba066be41546f5b540457cf2bddcec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:52:56.350746 2100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b6d61d84eb94287b268dbbd6c158c9e" candidate_uuid: "d965e6a28e4f46deb1de3fb7150a3607" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeba066be41546f5b540457cf2bddcec" is_pre_election: true
I20260928 22:52:56.350844 2298 leader_election.cc:302] T 2b6d61d84eb94287b268dbbd6c158c9e P d965e6a28e4f46deb1de3fb7150a3607 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d384f23e70ae46bc869ee4c7d3a08fae, d965e6a28e4f46deb1de3fb7150a3607; no voters:
I20260928 22:52:56.350947 2100 raft_consensus.cc:2454] T 2b6d61d84eb94287b268dbbd6c158c9e P aeba066be41546f5b540457cf2bddcec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d965e6a28e4f46deb1de3fb7150a3607 in term 0.
I20260928 22:52:56.350996 2430 raft_consensus.cc:2781] T 2b6d61d84eb94287b268dbbd6c158c9e P d965e6a28e4f46deb1de3fb7150a3607 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:52:56.351066 2430 raft_consensus.cc:482] T 2b6d61d84eb94287b268dbbd6c158c9e P d965e6a28e4f46deb1de3fb7150a3607 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:52:56.351105 2430 raft_consensus.cc:3037] T 2b6d61d84eb94287b268dbbd6c158c9e P d965e6a28e4f46deb1de3fb7150a3607 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:52:56.351867 2430 raft_consensus.cc:504] T 2b6d61d84eb94287b268dbbd6c158c9e P d965e6a28e4f46deb1de3fb7150a3607 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeba066be41546f5b540457cf2bddcec" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33291 } } peers { permanent_uuid: "d965e6a28e4f46deb1de3fb7150a3607" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41285 } } peers { permanent_uuid: "d384f23e70ae46bc869ee4c7d3a08fae" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 43421 } }
I20260928 22:52:56.352080 2430 leader_election.cc:288] T 2b6d61d84eb94287b268dbbd6c158c9e P d965e6a28e4f46deb1de3fb7150a3607 [CANDIDATE]: Term 1 election: Requested vote from peers aeba066be41546f5b540457cf2bddcec (127.16.121.129:33291), d384f23e70ae46bc869ee4c7d3a08fae (127.16.121.130:43421)
I20260928 22:52:56.352257 2100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b6d61d84eb94287b268dbbd6c158c9e" candidate_uuid: "d965e6a28e4f46deb1de3fb7150a3607" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeba066be41546f5b540457cf2bddcec"
I20260928 22:52:56.352242 2231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b6d61d84eb94287b268dbbd6c158c9e" candidate_uuid: "d965e6a28e4f46deb1de3fb7150a3607" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d384f23e70ae46bc869ee4c7d3a08fae"
I20260928 22:52:56.352339 2100 raft_consensus.cc:3037] T 2b6d61d84eb94287b268dbbd6c158c9e P aeba066be41546f5b540457cf2bddcec [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:52:56.352339 2231 raft_consensus.cc:3037] T 2b6d61d84eb94287b268dbbd6c158c9e P d384f23e70ae46bc869ee4c7d3a08fae [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:52:56.353291 2100 raft_consensus.cc:2454] T 2b6d61d84eb94287b268dbbd6c158c9e P aeba066be41546f5b540457cf2bddcec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d965e6a28e4f46deb1de3fb7150a3607 in term 1.
I20260928 22:52:56.353279 2231 raft_consensus.cc:2454] T 2b6d61d84eb94287b268dbbd6c158c9e P d384f23e70ae46bc869ee4c7d3a08fae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d965e6a28e4f46deb1de3fb7150a3607 in term 1.
I20260928 22:52:56.353502 2298 leader_election.cc:302] T 2b6d61d84eb94287b268dbbd6c158c9e P d965e6a28e4f46deb1de3fb7150a3607 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d384f23e70ae46bc869ee4c7d3a08fae, d965e6a28e4f46deb1de3fb7150a3607; no voters:
I20260928 22:52:56.353591 2430 raft_consensus.cc:2781] T 2b6d61d84eb94287b268dbbd6c158c9e P d965e6a28e4f46deb1de3fb7150a3607 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:52:56.353747 2430 raft_consensus.cc:686] T 2b6d61d84eb94287b268dbbd6c158c9e P d965e6a28e4f46deb1de3fb7150a3607 [term 1 LEADER]: Becoming Leader. State: Replica: d965e6a28e4f46deb1de3fb7150a3607, State: Running, Role: LEADER
I20260928 22:52:56.353843 2430 consensus_queue.cc:237] T 2b6d61d84eb94287b268dbbd6c158c9e P d965e6a28e4f46deb1de3fb7150a3607 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeba066be41546f5b540457cf2bddcec" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33291 } } peers { permanent_uuid: "d965e6a28e4f46deb1de3fb7150a3607" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41285 } } peers { permanent_uuid: "d384f23e70ae46bc869ee4c7d3a08fae" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 43421 } }
I20260928 22:52:56.354854 1957 catalog_manager.cc:5495] T 2b6d61d84eb94287b268dbbd6c158c9e P d965e6a28e4f46deb1de3fb7150a3607 reported cstate change: term changed from 0 to 1, leader changed from <none> to d965e6a28e4f46deb1de3fb7150a3607 (127.16.121.131). New cstate: current_term: 1 leader_uuid: "d965e6a28e4f46deb1de3fb7150a3607" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aeba066be41546f5b540457cf2bddcec" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d965e6a28e4f46deb1de3fb7150a3607" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d384f23e70ae46bc869ee4c7d3a08fae" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 43421 } health_report { overall_health: UNKNOWN } } }
W20260928 22:52:56.363510 2409 tablet.cc:2369] T 2b6d61d84eb94287b268dbbd6c158c9e P d965e6a28e4f46deb1de3fb7150a3607: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:52:56.389477 2419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:56.386681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40572 (local address 127.16.121.131:41285)
0928 22:52:56.386768 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:56.386770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:56.386778 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:52:56.386817 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:56.386817 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:56.386839 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:56.386904 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:52:56.387207 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:56.387695 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:56.388564 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:56.388813 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:56.388900 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:56.389094 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:52:56.389207 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:52:56.389310 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:56.389329 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 22:52:56.389354 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 22:52:56.391611 2231 raft_consensus.cc:1261] T 2b6d61d84eb94287b268dbbd6c158c9e P d384f23e70ae46bc869ee4c7d3a08fae [term 1 FOLLOWER]: Refusing update from remote peer d965e6a28e4f46deb1de3fb7150a3607: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:52:56.391611 2100 raft_consensus.cc:1261] T 2b6d61d84eb94287b268dbbd6c158c9e P aeba066be41546f5b540457cf2bddcec [term 1 FOLLOWER]: Refusing update from remote peer d965e6a28e4f46deb1de3fb7150a3607: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:52:56.392043 2432 consensus_queue.cc:1035] T 2b6d61d84eb94287b268dbbd6c158c9e P d965e6a28e4f46deb1de3fb7150a3607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeba066be41546f5b540457cf2bddcec" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:52:56.392303 2430 consensus_queue.cc:1035] T 2b6d61d84eb94287b268dbbd6c158c9e P d965e6a28e4f46deb1de3fb7150a3607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d384f23e70ae46bc869ee4c7d3a08fae" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 43421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:52:56.395205 2436 mvcc.cc:205] Tried to move back new op lower bound from 7334444959296659456 to 7334444959146205184. Current Snapshot: MvccSnapshot[applied={T|T < 7334444959296659456}]
I20260928 22:52:56.395325 2435 mvcc.cc:205] Tried to move back new op lower bound from 7334444959296659456 to 7334444959146205184. Current Snapshot: MvccSnapshot[applied={T|T < 7334444959296659456}]
I20260928 22:52:56.395444 2437 mvcc.cc:205] Tried to move back new op lower bound from 7334444959296659456 to 7334444959146205184. Current Snapshot: MvccSnapshot[applied={T|T < 7334444959296659456}]
I20260928 22:52:56.398013 1957 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:52:56.398137 1957 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:52:56.399217 1957 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b2de88f296b348b5b7a50be086bfa597: Sending DeleteTablet for 3 replicas of tablet 2b6d61d84eb94287b268dbbd6c158c9e
I20260928 22:52:56.399627 2211 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b6d61d84eb94287b268dbbd6c158c9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:52:56 UTC) from {username='slave'} at 127.0.0.1:33442
I20260928 22:52:56.399644 2080 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b6d61d84eb94287b268dbbd6c158c9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:52:56 UTC) from {username='slave'} at 127.0.0.1:54510
I20260928 22:52:56.399933 2446 tablet_replica.cc:330] T 2b6d61d84eb94287b268dbbd6c158c9e P aeba066be41546f5b540457cf2bddcec: stopping tablet replica
I20260928 22:52:56.400048 2446 raft_consensus.cc:2229] T 2b6d61d84eb94287b268dbbd6c158c9e P aeba066be41546f5b540457cf2bddcec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:52:56.400154 2446 raft_consensus.cc:2258] T 2b6d61d84eb94287b268dbbd6c158c9e P aeba066be41546f5b540457cf2bddcec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:52:56.400104 2342 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b6d61d84eb94287b268dbbd6c158c9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:52:56 UTC) from {username='slave'} at 127.0.0.1:40560
I20260928 22:52:56.400362 2447 tablet_replica.cc:330] T 2b6d61d84eb94287b268dbbd6c158c9e P d965e6a28e4f46deb1de3fb7150a3607: stopping tablet replica
I20260928 22:52:56.400439 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 2018
I20260928 22:52:56.400465 2447 raft_consensus.cc:2229] T 2b6d61d84eb94287b268dbbd6c158c9e P d965e6a28e4f46deb1de3fb7150a3607 [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:52:56.400556 2447 raft_consensus.cc:2258] T 2b6d61d84eb94287b268dbbd6c158c9e P d965e6a28e4f46deb1de3fb7150a3607 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:52:56.401388 2447 ts_tablet_manager.cc:1905] T 2b6d61d84eb94287b268dbbd6c158c9e P d965e6a28e4f46deb1de3fb7150a3607: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:52:56.403220 2447 ts_tablet_manager.cc:1918] T 2b6d61d84eb94287b268dbbd6c158c9e P d965e6a28e4f46deb1de3fb7150a3607: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:52:56.403333 2447 log.cc:1196] T 2b6d61d84eb94287b268dbbd6c158c9e P d965e6a28e4f46deb1de3fb7150a3607: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-2/wal/wals/2b6d61d84eb94287b268dbbd6c158c9e
I20260928 22:52:56.403415 2445 tablet_replica.cc:330] T 2b6d61d84eb94287b268dbbd6c158c9e P d384f23e70ae46bc869ee4c7d3a08fae: stopping tablet replica
I20260928 22:52:56.403558 2445 raft_consensus.cc:2229] T 2b6d61d84eb94287b268dbbd6c158c9e P d384f23e70ae46bc869ee4c7d3a08fae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:52:56.403604 2447 ts_tablet_manager.cc:1939] T 2b6d61d84eb94287b268dbbd6c158c9e P d965e6a28e4f46deb1de3fb7150a3607: Deleting consensus metadata
I20260928 22:52:56.403681 2445 raft_consensus.cc:2258] T 2b6d61d84eb94287b268dbbd6c158c9e P d384f23e70ae46bc869ee4c7d3a08fae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:52:56.404019 1943 catalog_manager.cc:4841] TS d965e6a28e4f46deb1de3fb7150a3607 (127.16.121.131:41285): tablet 2b6d61d84eb94287b268dbbd6c158c9e (table test-table [id=6c04791b548c401093098b4660d8c46c]) successfully deleted
I20260928 22:52:56.404183 2445 ts_tablet_manager.cc:1905] T 2b6d61d84eb94287b268dbbd6c158c9e P d384f23e70ae46bc869ee4c7d3a08fae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:52:56.406157 2445 ts_tablet_manager.cc:1918] T 2b6d61d84eb94287b268dbbd6c158c9e P d384f23e70ae46bc869ee4c7d3a08fae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:52:56.406244 2445 log.cc:1196] T 2b6d61d84eb94287b268dbbd6c158c9e P d384f23e70ae46bc869ee4c7d3a08fae: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790635733329745-16870-0/minicluster-data/ts-1/wal/wals/2b6d61d84eb94287b268dbbd6c158c9e
I20260928 22:52:56.406515 2445 ts_tablet_manager.cc:1939] T 2b6d61d84eb94287b268dbbd6c158c9e P d384f23e70ae46bc869ee4c7d3a08fae: Deleting consensus metadata
I20260928 22:52:56.406946 1945 catalog_manager.cc:4841] TS d384f23e70ae46bc869ee4c7d3a08fae (127.16.121.130:43421): tablet 2b6d61d84eb94287b268dbbd6c158c9e (table test-table [id=6c04791b548c401093098b4660d8c46c]) successfully deleted
W20260928 22:52:56.406981 1942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.121.129:33291 (error 108)
I20260928 22:52:56.407598 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 2149
I20260928 22:52:56.407603 2417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:56.407441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:33291 (local address 127.0.0.1:54520)
0928 22:52:56.407511 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:56.407563 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.129:33291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 22:52:56.408607 1942 catalog_manager.cc:4584] TS aeba066be41546f5b540457cf2bddcec (127.16.121.129:33291): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2b6d61d84eb94287b268dbbd6c158c9e: Network error: Client connection negotiation failed: client connection to 127.16.121.129:33291: connect: Connection refused (error 111)
I20260928 22:52:56.412626 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 2280
I20260928 22:52:56.417984 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 1927
2026-09-28T22:52:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (888 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:52:56 dist-test-slave-56vw krb5kdc[2451](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:52:56 dist-test-slave-56vw krb5kdc[2451](info): set up 2 sockets
Sep 28 22:52:56 dist-test-slave-56vw krb5kdc[2451](info): commencing operation
krb5kdc: starting...
W20260928 22:52:57.495124 16870 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Sep 28 22:52:57 dist-test-slave-56vw krb5kdc[2451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:52:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:52:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.121.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
I20260928 22:52:57.623467 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:37905
--webserver_interface=127.16.121.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:45459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:37905
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:52:57.710108 2464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:52:57.710353 2464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:52:57.710378 2464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:52:57.711648 2464 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:52:57.711696 2464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:52:57.711711 2464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:52:57.711725 2464 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:52:57.711737 2464 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:52:57.713299 2464 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:45459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:37905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:37905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.121.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:52:57.713510 2464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:52:57.713702 2464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 22:52:57.719030 2464 server_base.cc:1034] running on GCE node
W20260928 22:52:57.719110 2472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:52:57.718998 2469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:52:57.718986 2470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:52:57.719486 2464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:52:57.719774 2464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:52:57.720927 2464 hybrid_clock.cc:648] HybridClock initialized: now 1790635977720897 us; error 55 us; skew 500 ppm
Sep 28 22:52:57 dist-test-slave-56vw krb5kdc[2451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635977, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:52:57.724577 2464 init.cc:377] Logged in from keytab as kudu/127.16.121.190@KRBTEST.COM (short username kudu)
I20260928 22:52:57.725231 2464 webserver.cc:460] Webserver started at http://127.16.121.190:37301/ using document root <none> and password file <none>
I20260928 22:52:57.725481 2464 fs_manager.cc:359] Metadata directory not provided
I20260928 22:52:57.725530 2464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:52:57.725652 2464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:52:57.726850 2464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "4624131371ba4a9ea13f36cd80f92acd"
format_stamp: "Formatted at 2026-09-28 22:52:57 on dist-test-slave-56vw"
server_key: "6a5c9479cdb42d25e50debc32474a325"
server_key_iv: "5f1c113ed6296d2050e4b638d6fba669"
server_key_version: "encryptionkey@0"
I20260928 22:52:57.727299 2464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "4624131371ba4a9ea13f36cd80f92acd"
format_stamp: "Formatted at 2026-09-28 22:52:57 on dist-test-slave-56vw"
server_key: "6a5c9479cdb42d25e50debc32474a325"
server_key_iv: "5f1c113ed6296d2050e4b638d6fba669"
server_key_version: "encryptionkey@0"
I20260928 22:52:57.728761 2464 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:52:57.729521 2479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:57.729830 2464 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:52:57.729923 2464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "4624131371ba4a9ea13f36cd80f92acd"
format_stamp: "Formatted at 2026-09-28 22:52:57 on dist-test-slave-56vw"
server_key: "6a5c9479cdb42d25e50debc32474a325"
server_key_iv: "5f1c113ed6296d2050e4b638d6fba669"
server_key_version: "encryptionkey@0"
I20260928 22:52:57.729984 2464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:52:57.753902 2464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:52:57.756407 2464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:52:57.756536 2464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:52:57.761075 2464 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.190:37905
I20260928 22:52:57.761662 2464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:52:57.761960 2531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.190:37905 every 8 connection(s)
I20260928 22:52:57.762620 2532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:52:57.765518 2532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4624131371ba4a9ea13f36cd80f92acd: Bootstrap starting.
I20260928 22:52:57.766165 2532 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4624131371ba4a9ea13f36cd80f92acd: Neither blocks nor log segments found. Creating new log.
I20260928 22:52:57.766523 2532 log.cc:824] T 00000000000000000000000000000000 P 4624131371ba4a9ea13f36cd80f92acd: Log is configured to *not* fsync() on all Append() calls
I20260928 22:52:57.767724 2532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4624131371ba4a9ea13f36cd80f92acd: No bootstrap required, opened a new log
I20260928 22:52:57.769397 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 2464
I20260928 22:52:57.769531 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:52:57.769775 16870 external_mini_cluster.cc:1428] Setting key 4076be53e79e070fcf27c1e90e5e890f
I20260928 22:52:57.770533 2532 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4624131371ba4a9ea13f36cd80f92acd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4624131371ba4a9ea13f36cd80f92acd" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37905 } }
I20260928 22:52:57.770771 2532 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4624131371ba4a9ea13f36cd80f92acd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:52:57.770823 2532 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4624131371ba4a9ea13f36cd80f92acd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4624131371ba4a9ea13f36cd80f92acd, State: Initialized, Role: FOLLOWER
I20260928 22:52:57.770953 2532 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4624131371ba4a9ea13f36cd80f92acd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4624131371ba4a9ea13f36cd80f92acd" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37905 } }
I20260928 22:52:57.771056 2532 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4624131371ba4a9ea13f36cd80f92acd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:52:57.771297 2532 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4624131371ba4a9ea13f36cd80f92acd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:52:57.771410 2532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4624131371ba4a9ea13f36cd80f92acd [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:52:57.772629 2532 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4624131371ba4a9ea13f36cd80f92acd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4624131371ba4a9ea13f36cd80f92acd" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37905 } }
I20260928 22:52:57.772740 2532 leader_election.cc:302] T 00000000000000000000000000000000 P 4624131371ba4a9ea13f36cd80f92acd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4624131371ba4a9ea13f36cd80f92acd; no voters:
I20260928 22:52:57.772928 2532 leader_election.cc:288] T 00000000000000000000000000000000 P 4624131371ba4a9ea13f36cd80f92acd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:52:57.773167 2538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4624131371ba4a9ea13f36cd80f92acd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:52:57.773278 2532 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4624131371ba4a9ea13f36cd80f92acd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:52:57.773519 2538 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4624131371ba4a9ea13f36cd80f92acd [term 1 LEADER]: Becoming Leader. State: Replica: 4624131371ba4a9ea13f36cd80f92acd, State: Running, Role: LEADER
I20260928 22:52:57.773687 2538 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4624131371ba4a9ea13f36cd80f92acd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4624131371ba4a9ea13f36cd80f92acd" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37905 } }
I20260928 22:52:57.775098 2538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4624131371ba4a9ea13f36cd80f92acd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4624131371ba4a9ea13f36cd80f92acd. Latest consensus state: current_term: 1 leader_uuid: "4624131371ba4a9ea13f36cd80f92acd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4624131371ba4a9ea13f36cd80f92acd" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37905 } } }
I20260928 22:52:57.775214 2538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4624131371ba4a9ea13f36cd80f92acd [sys.catalog]: This master's current role is: LEADER
I20260928 22:52:57.775295 2537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4624131371ba4a9ea13f36cd80f92acd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4624131371ba4a9ea13f36cd80f92acd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4624131371ba4a9ea13f36cd80f92acd" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37905 } } }
I20260928 22:52:57.775365 2537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4624131371ba4a9ea13f36cd80f92acd [sys.catalog]: This master's current role is: LEADER
W20260928 22:52:57.776585 2552 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4624131371ba4a9ea13f36cd80f92acd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:52:57.776647 2552 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:52:57.776716 2551 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:52:57.776894 2551 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 22:52:57 dist-test-slave-56vw krb5kdc[2451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:52:57.779135 2551 catalog_manager.cc:1347] Generated new cluster ID: 72f4afc5d1d34763a58c05919ae7549d
I20260928 22:52:57.779192 2551 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:52:57.782951 2536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:57.770606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59774 (local address 127.16.121.190:37905)
0928 22:52:57.770910 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:57.770920 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:57.771090 (+ 170us) server_negotiation.cc:407] Connection header received
0928 22:52:57.771247 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:57.771255 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:57.771332 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:57.771478 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:52:57.772406 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:57.773854 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:57.775084 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:57.775877 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:57.775993 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:57.778961 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:52:57.778965 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:52:57.778966 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:52:57.778979 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:52:57.781019 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:52:57.781400 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:52:57.781403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:52:57.781403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:52:57.781525 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:52:57.781708 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:52:57.781709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:52:57.781710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:52:57.782012 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:52:57.782140 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:57.782605 (+ 465us) server_negotiation.cc:299] Negotiation successful
0928 22:52:57.782754 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":119,"threads_started":1}
I20260928 22:52:57.792444 2551 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:52:57.793049 2551 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:52:57.799364 2551 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4624131371ba4a9ea13f36cd80f92acd: Generated new TSK 0
I20260928 22:52:57.799604 2551 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.121.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab.
Entry for principal kudu/127.16.121.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab.
I20260928 22:52:57.838070 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.129:0
--local_ip_for_outbound_sockets=127.16.121.129
--webserver_interface=127.16.121.129
--webserver_port=0
--tserver_master_addrs=127.16.121.190:37905
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab
--principal=kudu/127.16.121.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:45459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:52:57.925768 2558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:52:57.925987 2558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:52:57.926012 2558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:52:57.927250 2558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:52:57.927292 2558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:52:57.927320 2558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.129
I20260928 22:52:57.928741 2558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:45459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab
--principal=kudu/127.16.121.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.121.129
--webserver_port=0
--tserver_master_addrs=127.16.121.190:37905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.129
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:52:57.928944 2558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:52:57.929122 2558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:52:57.934474 2564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:52:57.934566 2558 server_base.cc:1034] running on GCE node
W20260928 22:52:57.934520 2566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:52:57.934648 2563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:52:57.934870 2558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:52:57.935214 2558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:52:57.936391 2558 hybrid_clock.cc:648] HybridClock initialized: now 1790635977936388 us; error 39 us; skew 500 ppm
Sep 28 22:52:57 dist-test-slave-56vw krb5kdc[2451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635977, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:52:57.938877 2558 init.cc:377] Logged in from keytab as kudu/127.16.121.129@KRBTEST.COM (short username kudu)
I20260928 22:52:57.939436 2558 webserver.cc:460] Webserver started at http://127.16.121.129:44981/ using document root <none> and password file <none>
I20260928 22:52:57.939675 2558 fs_manager.cc:359] Metadata directory not provided
I20260928 22:52:57.939752 2558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:52:57.939904 2558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:52:57.941105 2558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-0/data/instance:
uuid: "e3fc2ac9ef89404bbff1cc8ad5090402"
format_stamp: "Formatted at 2026-09-28 22:52:57 on dist-test-slave-56vw"
server_key: "990f0adb8f29b8064682388aab362574"
server_key_iv: "9966fc93b446d48754109325abc94181"
server_key_version: "encryptionkey@0"
I20260928 22:52:57.941504 2558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance:
uuid: "e3fc2ac9ef89404bbff1cc8ad5090402"
format_stamp: "Formatted at 2026-09-28 22:52:57 on dist-test-slave-56vw"
server_key: "990f0adb8f29b8064682388aab362574"
server_key_iv: "9966fc93b446d48754109325abc94181"
server_key_version: "encryptionkey@0"
I20260928 22:52:57.942945 2558 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:52:57.943938 2573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:57.944164 2558 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:52:57.944291 2558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-0/wal
uuid: "e3fc2ac9ef89404bbff1cc8ad5090402"
format_stamp: "Formatted at 2026-09-28 22:52:57 on dist-test-slave-56vw"
server_key: "990f0adb8f29b8064682388aab362574"
server_key_iv: "9966fc93b446d48754109325abc94181"
server_key_version: "encryptionkey@0"
I20260928 22:52:57.944383 2558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:52:57.976608 2558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:52:57.979105 2558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:52:57.979310 2558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:52:57.979576 2558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:52:57.979902 2558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:52:57.979956 2558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:57.979997 2558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:52:57.980039 2558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:57.986382 2558 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.129:33813
I20260928 22:52:57.986440 2686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.129:33813 every 8 connection(s)
I20260928 22:52:57.986835 2558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
Sep 28 22:52:57 dist-test-slave-56vw krb5kdc[2451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635977, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:52:57.994619 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 2558
I20260928 22:52:57.994727 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance
I20260928 22:52:57.994979 16870 external_mini_cluster.cc:1428] Setting key b32520f1a503922c6ca812a0811c0f5e
I20260928 22:52:57.997470 2536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:57.987710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:41463 (local address 127.16.121.190:37905)
0928 22:52:57.987817 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:57.987819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:57.987915 (+ 96us) server_negotiation.cc:407] Connection header received
0928 22:52:57.988426 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:57.988428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:57.988441 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:57.988493 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:52:57.989503 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:57.990098 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:57.991292 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:57.991493 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:57.991616 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:57.993913 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:52:57.993914 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:52:57.993914 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:52:57.993921 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:52:57.995672 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:52:57.996252 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:52:57.996253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:52:57.996254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:52:57.996315 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:52:57.996626 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:52:57.996627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:52:57.996627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:52:57.996747 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:52:57.996810 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:57.997238 (+ 428us) server_negotiation.cc:299] Negotiation successful
0928 22:52:57.997309 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 22:52:57.997768 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:57.987623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37905 (local address 127.16.121.129:41463)
0928 22:52:57.987801 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:57.987840 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 22:52:57.988337 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:52:57.988553 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:52:57.988556 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:52:57.989181 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:52:57.989386 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:57.989392 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:57.990216 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:57.990217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:57.991194 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:57.991197 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:57.991634 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:57.991635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:57.991751 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:57.992413 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:52:57.992422 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:52:57.993821 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:52:57.995770 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:52:57.995779 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:52:57.995781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:52:57.996163 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:52:57.996392 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:52:57.996393 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:52:57.996394 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:52:57.996550 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:52:57.996873 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:52:57.996881 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:52:57.997072 (+ 191us) client_negotiation.cc:769] Sending connection context
0928 22:52:57.997197 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 22:52:57.997383 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":79,"threads_started":1}
I20260928 22:52:57.998337 2687 heartbeater.cc:347] Connected to a master server at 127.16.121.190:37905
I20260928 22:52:57.998451 2687 heartbeater.cc:464] Registering TS with master...
I20260928 22:52:57.998662 2687 heartbeater.cc:511] Master 127.16.121.190:37905 requested a full tablet report, sending...
I20260928 22:52:57.999349 2496 ts_manager.cc:194] Registered new tserver with Master: e3fc2ac9ef89404bbff1cc8ad5090402 (127.16.121.129:33813)
I20260928 22:52:58.000309 2496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.129@KRBTEST.COM'} at 127.16.121.129:41463
WARNING: no policy specified for kudu/127.16.121.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab.
Entry for principal kudu/127.16.121.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab.
I20260928 22:52:58.023293 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.130:0
--local_ip_for_outbound_sockets=127.16.121.130
--webserver_interface=127.16.121.130
--webserver_port=0
--tserver_master_addrs=127.16.121.190:37905
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab
--principal=kudu/127.16.121.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:45459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:52:58.110718 2692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:52:58.110975 2692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:52:58.111027 2692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:52:58.112280 2692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:52:58.112349 2692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:52:58.112394 2692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.130
I20260928 22:52:58.113899 2692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:45459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab
--principal=kudu/127.16.121.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.121.130
--webserver_port=0
--tserver_master_addrs=127.16.121.190:37905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.130
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:52:58.114185 2692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:52:58.114421 2692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:52:58.120002 2700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:52:58.120034 2697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:52:58.120046 2698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:52:58.120442 2692 server_base.cc:1034] running on GCE node
I20260928 22:52:58.120718 2692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:52:58.120990 2692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:52:58.122166 2692 hybrid_clock.cc:648] HybridClock initialized: now 1790635978122142 us; error 43 us; skew 500 ppm
Sep 28 22:52:58 dist-test-slave-56vw krb5kdc[2451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635978, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:52:58.125485 2692 init.cc:377] Logged in from keytab as kudu/127.16.121.130@KRBTEST.COM (short username kudu)
I20260928 22:52:58.126017 2692 webserver.cc:460] Webserver started at http://127.16.121.130:43871/ using document root <none> and password file <none>
I20260928 22:52:58.126261 2692 fs_manager.cc:359] Metadata directory not provided
I20260928 22:52:58.126333 2692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:52:58.126503 2692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:52:58.127730 2692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-1/data/instance:
uuid: "7ba2ad8264ef43c7b3539785d4a67533"
format_stamp: "Formatted at 2026-09-28 22:52:58 on dist-test-slave-56vw"
server_key: "f651cb2d966f8047b7c0532f19cab774"
server_key_iv: "bb95c9a03993b5f4d0b7fca71a4f74af"
server_key_version: "encryptionkey@0"
I20260928 22:52:58.128180 2692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance:
uuid: "7ba2ad8264ef43c7b3539785d4a67533"
format_stamp: "Formatted at 2026-09-28 22:52:58 on dist-test-slave-56vw"
server_key: "f651cb2d966f8047b7c0532f19cab774"
server_key_iv: "bb95c9a03993b5f4d0b7fca71a4f74af"
server_key_version: "encryptionkey@0"
I20260928 22:52:58.129806 2692 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:52:58.130913 2707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:58.131224 2692 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:52:58.131335 2692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-1/wal
uuid: "7ba2ad8264ef43c7b3539785d4a67533"
format_stamp: "Formatted at 2026-09-28 22:52:58 on dist-test-slave-56vw"
server_key: "f651cb2d966f8047b7c0532f19cab774"
server_key_iv: "bb95c9a03993b5f4d0b7fca71a4f74af"
server_key_version: "encryptionkey@0"
I20260928 22:52:58.131431 2692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:52:58.141170 2692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:52:58.143687 2692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:52:58.143873 2692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:52:58.144169 2692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:52:58.144505 2692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:52:58.144560 2692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:58.144600 2692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:52:58.144660 2692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:58.150729 2692 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.130:38019
I20260928 22:52:58.150841 2820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.130:38019 every 8 connection(s)
I20260928 22:52:58.151180 2692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
Sep 28 22:52:58 dist-test-slave-56vw krb5kdc[2451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635978, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.130@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:52:58.158635 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 2692
I20260928 22:52:58.158769 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance
I20260928 22:52:58.159067 16870 external_mini_cluster.cc:1428] Setting key dc7be107bc45aa6d9dea790533e09d5e
I20260928 22:52:58.161556 2536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:58.151962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:36845 (local address 127.16.121.190:37905)
0928 22:52:58.152012 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:58.152014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:58.152191 (+ 177us) server_negotiation.cc:407] Connection header received
0928 22:52:58.152705 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:58.152706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:58.152715 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:58.152784 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:52:58.153712 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:58.154210 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:58.155313 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:58.155530 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:58.155594 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:58.157993 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:52:58.157995 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:52:58.157995 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:52:58.158003 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:52:58.160003 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:52:58.160456 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:52:58.160457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:52:58.160458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:52:58.160525 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:52:58.160821 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:52:58.160822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:52:58.160822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:52:58.160945 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:52:58.161017 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:58.161391 (+ 374us) server_negotiation.cc:299] Negotiation successful
0928 22:52:58.161472 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:52:58.161840 2823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:58.151913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37905 (local address 127.16.121.130:36845)
0928 22:52:58.152072 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:58.152117 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 22:52:58.152603 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:52:58.152843 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:52:58.152846 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:52:58.153416 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:52:58.153623 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:58.153627 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:58.154291 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:58.154292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:58.155227 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:58.155230 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:58.155638 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:58.155639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:58.155769 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:58.156308 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:52:58.156317 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:52:58.157878 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:52:58.160094 (+ 2216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:52:58.160103 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:52:58.160106 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:52:58.160359 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:52:58.160609 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:52:58.160610 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:52:58.160610 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:52:58.160736 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:52:58.161019 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:52:58.161028 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:52:58.161227 (+ 199us) client_negotiation.cc:769] Sending connection context
0928 22:52:58.161359 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 22:52:58.161557 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
I20260928 22:52:58.162331 2821 heartbeater.cc:347] Connected to a master server at 127.16.121.190:37905
I20260928 22:52:58.162446 2821 heartbeater.cc:464] Registering TS with master...
I20260928 22:52:58.162711 2821 heartbeater.cc:511] Master 127.16.121.190:37905 requested a full tablet report, sending...
I20260928 22:52:58.163233 2496 ts_manager.cc:194] Registered new tserver with Master: 7ba2ad8264ef43c7b3539785d4a67533 (127.16.121.130:38019)
I20260928 22:52:58.163630 2496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.130@KRBTEST.COM'} at 127.16.121.130:36845
WARNING: no policy specified for kudu/127.16.121.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab.
Entry for principal kudu/127.16.121.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab.
I20260928 22:52:58.187253 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.131:0
--local_ip_for_outbound_sockets=127.16.121.131
--webserver_interface=127.16.121.131
--webserver_port=0
--tserver_master_addrs=127.16.121.190:37905
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab
--principal=kudu/127.16.121.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:45459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:52:58.271966 2826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:52:58.272199 2826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:52:58.272224 2826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:52:58.273511 2826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:52:58.273557 2826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:52:58.273586 2826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.131
I20260928 22:52:58.275305 2826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:45459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab
--principal=kudu/127.16.121.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.121.131
--webserver_port=0
--tserver_master_addrs=127.16.121.190:37905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.131
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:52:58.275578 2826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:52:58.275826 2826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:52:58.281509 2832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:52:58.281543 2831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:52:58.281625 2826 server_base.cc:1034] running on GCE node
W20260928 22:52:58.281545 2834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:52:58.281987 2826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:52:58.282266 2826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:52:58.283432 2826 hybrid_clock.cc:648] HybridClock initialized: now 1790635978283429 us; error 35 us; skew 500 ppm
Sep 28 22:52:58 dist-test-slave-56vw krb5kdc[2451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635978, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:52:58.286002 2826 init.cc:377] Logged in from keytab as kudu/127.16.121.131@KRBTEST.COM (short username kudu)
I20260928 22:52:58.286490 2826 webserver.cc:460] Webserver started at http://127.16.121.131:44245/ using document root <none> and password file <none>
I20260928 22:52:58.286718 2826 fs_manager.cc:359] Metadata directory not provided
I20260928 22:52:58.286788 2826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:52:58.286939 2826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:52:58.288026 2826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-2/data/instance:
uuid: "68e273fb4ee048eeac9848cb05bfa17b"
format_stamp: "Formatted at 2026-09-28 22:52:58 on dist-test-slave-56vw"
server_key: "7db9e96a496b6a05023a2e1fce6da1bf"
server_key_iv: "fcb3a08d7a04421f5d2412cbf7a7e44b"
server_key_version: "encryptionkey@0"
I20260928 22:52:58.288396 2826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance:
uuid: "68e273fb4ee048eeac9848cb05bfa17b"
format_stamp: "Formatted at 2026-09-28 22:52:58 on dist-test-slave-56vw"
server_key: "7db9e96a496b6a05023a2e1fce6da1bf"
server_key_iv: "fcb3a08d7a04421f5d2412cbf7a7e44b"
server_key_version: "encryptionkey@0"
I20260928 22:52:58.289757 2826 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 22:52:58.290561 2841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:58.290776 2826 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:52:58.290875 2826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-2/wal
uuid: "68e273fb4ee048eeac9848cb05bfa17b"
format_stamp: "Formatted at 2026-09-28 22:52:58 on dist-test-slave-56vw"
server_key: "7db9e96a496b6a05023a2e1fce6da1bf"
server_key_iv: "fcb3a08d7a04421f5d2412cbf7a7e44b"
server_key_version: "encryptionkey@0"
I20260928 22:52:58.290959 2826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:52:58.309787 2826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:52:58.312417 2826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:52:58.312588 2826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:52:58.312834 2826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:52:58.313160 2826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:52:58.313217 2826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:58.313259 2826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:52:58.313302 2826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:58.320161 2826 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.131:43767
I20260928 22:52:58.320207 2954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.131:43767 every 8 connection(s)
I20260928 22:52:58.320600 2826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
I20260928 22:52:58.322472 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 2826
I20260928 22:52:58.322619 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance
I20260928 22:52:58.322805 16870 external_mini_cluster.cc:1428] Setting key 5793c3406341402f28100435e4478b95
Sep 28 22:52:58 dist-test-slave-56vw krb5kdc[2451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635978, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.131@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:52:58.331234 2536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:58.321413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:43339 (local address 127.16.121.190:37905)
0928 22:52:58.321539 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:58.321542 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:58.321720 (+ 178us) server_negotiation.cc:407] Connection header received
0928 22:52:58.322305 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:58.322306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:58.322320 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:58.322377 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:52:58.323709 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:58.324281 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:58.325333 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:58.325549 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:58.325623 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:58.328097 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:52:58.328099 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:52:58.328099 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:52:58.328107 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:52:58.329519 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:52:58.330075 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:52:58.330076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:52:58.330077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:52:58.330159 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:52:58.330516 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:52:58.330517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:52:58.330518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:52:58.330653 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:52:58.330735 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:58.331024 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 22:52:58.331132 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260928 22:52:58.331465 2957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:58.321359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37905 (local address 127.16.121.131:43339)
0928 22:52:58.321571 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:58.321621 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 22:52:58.322150 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:52:58.322451 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:52:58.322456 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:52:58.323339 (+ 883us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:52:58.323631 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:58.323639 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:58.324375 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:58.324376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:58.325234 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:58.325236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:58.325644 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:58.325645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:58.325793 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:58.326408 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:52:58.326417 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:52:58.327990 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:52:58.329713 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:52:58.329721 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:52:58.329724 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:52:58.329988 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:52:58.330233 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:52:58.330234 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:52:58.330234 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:52:58.330424 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:52:58.330725 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:52:58.330733 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:52:58.330881 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 22:52:58.331004 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 22:52:58.331210 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":90,"threads_started":1}
I20260928 22:52:58.332005 2955 heartbeater.cc:347] Connected to a master server at 127.16.121.190:37905
I20260928 22:52:58.332105 2955 heartbeater.cc:464] Registering TS with master...
I20260928 22:52:58.332260 2955 heartbeater.cc:511] Master 127.16.121.190:37905 requested a full tablet report, sending...
I20260928 22:52:58.332715 2496 ts_manager.cc:194] Registered new tserver with Master: 68e273fb4ee048eeac9848cb05bfa17b (127.16.121.131:43767)
I20260928 22:52:58.333097 2496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.131@KRBTEST.COM'} at 127.16.121.131:43339
I20260928 22:52:58.335114 16870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:52:58.343108 2536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:58.336613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48342 (local address 127.16.121.190:37905)
0928 22:52:58.336655 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:58.336657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:58.336826 (+ 169us) server_negotiation.cc:407] Connection header received
0928 22:52:58.336904 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:58.336905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:58.336915 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:58.336973 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:52:58.337540 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:58.338042 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:58.338819 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:58.339016 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:58.339132 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:58.340585 (+ 1453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:52:58.340586 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:52:58.340587 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:52:58.340595 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:52:58.341803 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:52:58.342206 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:52:58.342207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:52:58.342208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:52:58.342270 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:52:58.342497 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:52:58.342498 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:52:58.342498 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:52:58.342648 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:52:58.342769 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:58.342899 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 22:52:58.342967 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:52:58.344875 2496 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48342:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:52:58.346417 2496 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:52:58.353621 2970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:58.349411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40654 (local address 127.16.121.130:38019)
0928 22:52:58.349690 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:58.349696 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:58.349709 (+ 13us) server_negotiation.cc:407] Connection header received
0928 22:52:58.349778 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:58.349780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:58.349797 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:58.349875 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:52:58.350137 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:58.350876 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:58.352377 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:58.353141 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:58.353221 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:58.353288 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:58.353479 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 22:52:58.353525 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":77,"threads_started":1}
I20260928 22:52:58.353615 2967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:58.349401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:33813 (local address 127.0.0.1:47264)
0928 22:52:58.349798 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:58.349807 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:52:58.349858 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:52:58.350274 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:52:58.350275 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:52:58.350277 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:52:58.350326 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:58.350328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:58.351608 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:58.351609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:58.352605 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:58.352607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:58.353387 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:58.353388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:58.353447 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:58.353451 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:52:58.353495 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 22:52:58.353537 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":83,"threads_started":1}
I20260928 22:52:58.353929 2968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:58.349297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43174 (local address 127.16.121.131:43767)
0928 22:52:58.349797 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:58.349801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:58.349813 (+ 12us) server_negotiation.cc:407] Connection header received
0928 22:52:58.349975 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:58.349977 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:58.349995 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:58.350077 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:52:58.350198 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:58.351364 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:58.352708 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:58.353446 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:58.353517 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:58.353566 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:58.353829 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 22:52:58.353866 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":104,"threads_started":1}
I20260928 22:52:58.354029 2971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:58.349506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47264 (local address 127.16.121.129:33813)
0928 22:52:58.349711 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:58.349716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:58.349858 (+ 142us) server_negotiation.cc:407] Connection header received
0928 22:52:58.350085 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:58.350099 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:58.350114 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:58.350187 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:52:58.350406 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:58.351495 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:58.352697 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:58.353224 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:58.353299 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:58.353356 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:58.353903 (+ 547us) server_negotiation.cc:299] Negotiation successful
0928 22:52:58.353948 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":90,"threads_started":1}
I20260928 22:52:58.353615 2966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:58.349199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.130:38019 (local address 127.0.0.1:40654)
0928 22:52:58.349549 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:58.349596 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 22:52:58.349676 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:52:58.349946 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:52:58.349949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:52:58.349955 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:52:58.350030 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:58.350032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:58.350994 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:58.350995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:58.352265 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:58.352268 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:58.353250 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:58.353252 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:58.353353 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:58.353370 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 22:52:58.353446 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 22:52:58.353514 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":5,"thread_start_us":71,"threads_started":1}
I20260928 22:52:58.354635 2969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:58.349199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.131:43767 (local address 127.0.0.1:43174)
0928 22:52:58.349550 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:58.349602 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 22:52:58.349873 (+ 271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:52:58.350090 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:52:58.350091 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:52:58.350093 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:52:58.350137 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:58.350138 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:58.351478 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:58.351479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:58.352613 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:58.352616 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:58.353612 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:58.353613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:58.353681 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:58.353686 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:52:58.353737 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 22:52:58.353778 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":56,"threads_started":1}
I20260928 22:52:58.354813 2621 tablet_service.cc:1467] Processing CreateTablet for tablet bb399eef861f49f2b4d8a8197647e519 (DEFAULT_TABLE table=test-table [id=236df766a0984fe88e1e3a2bf36833a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:52:58.354933 2755 tablet_service.cc:1467] Processing CreateTablet for tablet bb399eef861f49f2b4d8a8197647e519 (DEFAULT_TABLE table=test-table [id=236df766a0984fe88e1e3a2bf36833a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:52:58.355237 2621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb399eef861f49f2b4d8a8197647e519. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:52:58.355237 2755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb399eef861f49f2b4d8a8197647e519. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:52:58.355409 2889 tablet_service.cc:1467] Processing CreateTablet for tablet bb399eef861f49f2b4d8a8197647e519 (DEFAULT_TABLE table=test-table [id=236df766a0984fe88e1e3a2bf36833a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:52:58.355744 2889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb399eef861f49f2b4d8a8197647e519. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:52:58.358356 2972 tablet_bootstrap.cc:492] T bb399eef861f49f2b4d8a8197647e519 P 68e273fb4ee048eeac9848cb05bfa17b: Bootstrap starting.
I20260928 22:52:58.359071 2973 tablet_bootstrap.cc:492] T bb399eef861f49f2b4d8a8197647e519 P 7ba2ad8264ef43c7b3539785d4a67533: Bootstrap starting.
I20260928 22:52:58.359200 2974 tablet_bootstrap.cc:492] T bb399eef861f49f2b4d8a8197647e519 P e3fc2ac9ef89404bbff1cc8ad5090402: Bootstrap starting.
I20260928 22:52:58.359256 2972 tablet_bootstrap.cc:654] T bb399eef861f49f2b4d8a8197647e519 P 68e273fb4ee048eeac9848cb05bfa17b: Neither blocks nor log segments found. Creating new log.
I20260928 22:52:58.359750 2974 tablet_bootstrap.cc:654] T bb399eef861f49f2b4d8a8197647e519 P e3fc2ac9ef89404bbff1cc8ad5090402: Neither blocks nor log segments found. Creating new log.
I20260928 22:52:58.359773 2973 tablet_bootstrap.cc:654] T bb399eef861f49f2b4d8a8197647e519 P 7ba2ad8264ef43c7b3539785d4a67533: Neither blocks nor log segments found. Creating new log.
I20260928 22:52:58.359890 2972 log.cc:824] T bb399eef861f49f2b4d8a8197647e519 P 68e273fb4ee048eeac9848cb05bfa17b: Log is configured to *not* fsync() on all Append() calls
I20260928 22:52:58.360379 2974 log.cc:824] T bb399eef861f49f2b4d8a8197647e519 P e3fc2ac9ef89404bbff1cc8ad5090402: Log is configured to *not* fsync() on all Append() calls
I20260928 22:52:58.360529 2973 log.cc:824] T bb399eef861f49f2b4d8a8197647e519 P 7ba2ad8264ef43c7b3539785d4a67533: Log is configured to *not* fsync() on all Append() calls
I20260928 22:52:58.360819 2972 tablet_bootstrap.cc:492] T bb399eef861f49f2b4d8a8197647e519 P 68e273fb4ee048eeac9848cb05bfa17b: No bootstrap required, opened a new log
I20260928 22:52:58.360904 2972 ts_tablet_manager.cc:1397] T bb399eef861f49f2b4d8a8197647e519 P 68e273fb4ee048eeac9848cb05bfa17b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 22:52:58.361326 2973 tablet_bootstrap.cc:492] T bb399eef861f49f2b4d8a8197647e519 P 7ba2ad8264ef43c7b3539785d4a67533: No bootstrap required, opened a new log
I20260928 22:52:58.361395 2973 ts_tablet_manager.cc:1397] T bb399eef861f49f2b4d8a8197647e519 P 7ba2ad8264ef43c7b3539785d4a67533: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:52:58.362967 2972 raft_consensus.cc:348] T bb399eef861f49f2b4d8a8197647e519 P 68e273fb4ee048eeac9848cb05bfa17b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68e273fb4ee048eeac9848cb05bfa17b" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 43767 } } peers { permanent_uuid: "e3fc2ac9ef89404bbff1cc8ad5090402" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33813 } } peers { permanent_uuid: "7ba2ad8264ef43c7b3539785d4a67533" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38019 } }
I20260928 22:52:58.363210 2972 raft_consensus.cc:374] T bb399eef861f49f2b4d8a8197647e519 P 68e273fb4ee048eeac9848cb05bfa17b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:52:58.363260 2972 raft_consensus.cc:729] T bb399eef861f49f2b4d8a8197647e519 P 68e273fb4ee048eeac9848cb05bfa17b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68e273fb4ee048eeac9848cb05bfa17b, State: Initialized, Role: FOLLOWER
I20260928 22:52:58.363422 2972 consensus_queue.cc:260] T bb399eef861f49f2b4d8a8197647e519 P 68e273fb4ee048eeac9848cb05bfa17b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68e273fb4ee048eeac9848cb05bfa17b" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 43767 } } peers { permanent_uuid: "e3fc2ac9ef89404bbff1cc8ad5090402" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33813 } } peers { permanent_uuid: "7ba2ad8264ef43c7b3539785d4a67533" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38019 } }
I20260928 22:52:58.363445 2973 raft_consensus.cc:348] T bb399eef861f49f2b4d8a8197647e519 P 7ba2ad8264ef43c7b3539785d4a67533 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68e273fb4ee048eeac9848cb05bfa17b" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 43767 } } peers { permanent_uuid: "e3fc2ac9ef89404bbff1cc8ad5090402" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33813 } } peers { permanent_uuid: "7ba2ad8264ef43c7b3539785d4a67533" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38019 } }
I20260928 22:52:58.363616 2973 raft_consensus.cc:374] T bb399eef861f49f2b4d8a8197647e519 P 7ba2ad8264ef43c7b3539785d4a67533 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:52:58.363667 2973 raft_consensus.cc:729] T bb399eef861f49f2b4d8a8197647e519 P 7ba2ad8264ef43c7b3539785d4a67533 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ba2ad8264ef43c7b3539785d4a67533, State: Initialized, Role: FOLLOWER
I20260928 22:52:58.363736 2972 ts_tablet_manager.cc:1428] T bb399eef861f49f2b4d8a8197647e519 P 68e273fb4ee048eeac9848cb05bfa17b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:52:58.363765 2974 tablet_bootstrap.cc:492] T bb399eef861f49f2b4d8a8197647e519 P e3fc2ac9ef89404bbff1cc8ad5090402: No bootstrap required, opened a new log
I20260928 22:52:58.363834 2974 ts_tablet_manager.cc:1397] T bb399eef861f49f2b4d8a8197647e519 P e3fc2ac9ef89404bbff1cc8ad5090402: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260928 22:52:58.363806 2973 consensus_queue.cc:260] T bb399eef861f49f2b4d8a8197647e519 P 7ba2ad8264ef43c7b3539785d4a67533 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68e273fb4ee048eeac9848cb05bfa17b" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 43767 } } peers { permanent_uuid: "e3fc2ac9ef89404bbff1cc8ad5090402" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33813 } } peers { permanent_uuid: "7ba2ad8264ef43c7b3539785d4a67533" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38019 } }
I20260928 22:52:58.363885 2955 heartbeater.cc:503] Master 127.16.121.190:37905 was elected leader, sending a full tablet report...
I20260928 22:52:58.364110 2821 heartbeater.cc:503] Master 127.16.121.190:37905 was elected leader, sending a full tablet report...
I20260928 22:52:58.364111 2973 ts_tablet_manager.cc:1428] T bb399eef861f49f2b4d8a8197647e519 P 7ba2ad8264ef43c7b3539785d4a67533: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:52:58.365757 2974 raft_consensus.cc:348] T bb399eef861f49f2b4d8a8197647e519 P e3fc2ac9ef89404bbff1cc8ad5090402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68e273fb4ee048eeac9848cb05bfa17b" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 43767 } } peers { permanent_uuid: "e3fc2ac9ef89404bbff1cc8ad5090402" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33813 } } peers { permanent_uuid: "7ba2ad8264ef43c7b3539785d4a67533" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38019 } }
I20260928 22:52:58.365923 2974 raft_consensus.cc:374] T bb399eef861f49f2b4d8a8197647e519 P e3fc2ac9ef89404bbff1cc8ad5090402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:52:58.365970 2974 raft_consensus.cc:729] T bb399eef861f49f2b4d8a8197647e519 P e3fc2ac9ef89404bbff1cc8ad5090402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fc2ac9ef89404bbff1cc8ad5090402, State: Initialized, Role: FOLLOWER
I20260928 22:52:58.366108 2974 consensus_queue.cc:260] T bb399eef861f49f2b4d8a8197647e519 P e3fc2ac9ef89404bbff1cc8ad5090402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68e273fb4ee048eeac9848cb05bfa17b" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 43767 } } peers { permanent_uuid: "e3fc2ac9ef89404bbff1cc8ad5090402" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33813 } } peers { permanent_uuid: "7ba2ad8264ef43c7b3539785d4a67533" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38019 } }
I20260928 22:52:58.366362 2974 ts_tablet_manager.cc:1428] T bb399eef861f49f2b4d8a8197647e519 P e3fc2ac9ef89404bbff1cc8ad5090402: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:52:58.366426 2687 heartbeater.cc:503] Master 127.16.121.190:37905 was elected leader, sending a full tablet report...
W20260928 22:52:58.401801 2822 tablet.cc:2369] T bb399eef861f49f2b4d8a8197647e519 P 7ba2ad8264ef43c7b3539785d4a67533: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:52:58.487679 2688 tablet.cc:2369] T bb399eef861f49f2b4d8a8197647e519 P e3fc2ac9ef89404bbff1cc8ad5090402: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:52:58.571240 2956 tablet.cc:2369] T bb399eef861f49f2b4d8a8197647e519 P 68e273fb4ee048eeac9848cb05bfa17b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:52:58.788534 2980 raft_consensus.cc:482] T bb399eef861f49f2b4d8a8197647e519 P e3fc2ac9ef89404bbff1cc8ad5090402 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:52:58.788677 2980 raft_consensus.cc:504] T bb399eef861f49f2b4d8a8197647e519 P e3fc2ac9ef89404bbff1cc8ad5090402 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68e273fb4ee048eeac9848cb05bfa17b" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 43767 } } peers { permanent_uuid: "e3fc2ac9ef89404bbff1cc8ad5090402" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33813 } } peers { permanent_uuid: "7ba2ad8264ef43c7b3539785d4a67533" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38019 } }
I20260928 22:52:58.789037 2980 leader_election.cc:288] T bb399eef861f49f2b4d8a8197647e519 P e3fc2ac9ef89404bbff1cc8ad5090402 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68e273fb4ee048eeac9848cb05bfa17b (127.16.121.131:43767), 7ba2ad8264ef43c7b3539785d4a67533 (127.16.121.130:38019)
I20260928 22:52:58.792233 2968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:58.789419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:34063 (local address 127.16.121.131:43767)
0928 22:52:58.789592 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:58.789594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:58.789603 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:52:58.789642 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:58.789643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:58.789650 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:58.789724 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:52:58.789923 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:58.790407 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:58.791506 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:58.791862 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:58.791961 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:58.791970 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:58.792122 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 22:52:58.792162 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260928 22:52:58.792250 2982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:58.789274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.131:43767 (local address 127.16.121.129:34063)
0928 22:52:58.789433 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:58.789446 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:52:58.789524 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:52:58.789729 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:52:58.789730 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:52:58.789732 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:52:58.789826 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:58.789828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:58.790518 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:58.790519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:58.791407 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:58.791409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:58.791964 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:58.791965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:58.792040 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:58.792044 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:52:58.792114 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 22:52:58.792156 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":72,"threads_started":1}
I20260928 22:52:58.792466 2970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:58.789371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:60653 (local address 127.16.121.130:38019)
0928 22:52:58.789426 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:58.789429 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:58.789600 (+ 171us) server_negotiation.cc:407] Connection header received
0928 22:52:58.789677 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:58.789678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:58.789693 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:58.789780 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:52:58.789981 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:58.790593 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:58.791711 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:58.792097 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:58.792191 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:58.792200 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:58.792368 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 22:52:58.792411 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 22:52:58.792536 2981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:58.789274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.130:38019 (local address 127.16.121.129:60653)
0928 22:52:58.789450 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:58.789463 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:52:58.789608 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:52:58.789785 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:52:58.789787 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:52:58.789789 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:52:58.789898 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:58.789901 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:58.790729 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:58.790731 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:58.791618 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:58.791620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:58.792189 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:58.792190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:58.792274 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:58.792277 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:52:58.792359 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 22:52:58.792415 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":19,"thread_start_us":73,"threads_started":1}
I20260928 22:52:58.792807 2909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb399eef861f49f2b4d8a8197647e519" candidate_uuid: "e3fc2ac9ef89404bbff1cc8ad5090402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68e273fb4ee048eeac9848cb05bfa17b" is_pre_election: true
I20260928 22:52:58.792879 2775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb399eef861f49f2b4d8a8197647e519" candidate_uuid: "e3fc2ac9ef89404bbff1cc8ad5090402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ba2ad8264ef43c7b3539785d4a67533" is_pre_election: true
I20260928 22:52:58.793013 2909 raft_consensus.cc:2454] T bb399eef861f49f2b4d8a8197647e519 P 68e273fb4ee048eeac9848cb05bfa17b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fc2ac9ef89404bbff1cc8ad5090402 in term 0.
I20260928 22:52:58.793044 2775 raft_consensus.cc:2454] T bb399eef861f49f2b4d8a8197647e519 P 7ba2ad8264ef43c7b3539785d4a67533 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fc2ac9ef89404bbff1cc8ad5090402 in term 0.
I20260928 22:52:58.793282 2577 leader_election.cc:302] T bb399eef861f49f2b4d8a8197647e519 P e3fc2ac9ef89404bbff1cc8ad5090402 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68e273fb4ee048eeac9848cb05bfa17b, e3fc2ac9ef89404bbff1cc8ad5090402; no voters:
I20260928 22:52:58.793483 2980 raft_consensus.cc:2781] T bb399eef861f49f2b4d8a8197647e519 P e3fc2ac9ef89404bbff1cc8ad5090402 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:52:58.793546 2980 raft_consensus.cc:482] T bb399eef861f49f2b4d8a8197647e519 P e3fc2ac9ef89404bbff1cc8ad5090402 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:52:58.793565 2980 raft_consensus.cc:3037] T bb399eef861f49f2b4d8a8197647e519 P e3fc2ac9ef89404bbff1cc8ad5090402 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:52:58.794435 2980 raft_consensus.cc:504] T bb399eef861f49f2b4d8a8197647e519 P e3fc2ac9ef89404bbff1cc8ad5090402 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68e273fb4ee048eeac9848cb05bfa17b" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 43767 } } peers { permanent_uuid: "e3fc2ac9ef89404bbff1cc8ad5090402" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33813 } } peers { permanent_uuid: "7ba2ad8264ef43c7b3539785d4a67533" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38019 } }
I20260928 22:52:58.794592 2980 leader_election.cc:288] T bb399eef861f49f2b4d8a8197647e519 P e3fc2ac9ef89404bbff1cc8ad5090402 [CANDIDATE]: Term 1 election: Requested vote from peers 68e273fb4ee048eeac9848cb05bfa17b (127.16.121.131:43767), 7ba2ad8264ef43c7b3539785d4a67533 (127.16.121.130:38019)
I20260928 22:52:58.794801 2909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb399eef861f49f2b4d8a8197647e519" candidate_uuid: "e3fc2ac9ef89404bbff1cc8ad5090402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68e273fb4ee048eeac9848cb05bfa17b"
I20260928 22:52:58.794888 2909 raft_consensus.cc:3037] T bb399eef861f49f2b4d8a8197647e519 P 68e273fb4ee048eeac9848cb05bfa17b [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:52:58.794907 2775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb399eef861f49f2b4d8a8197647e519" candidate_uuid: "e3fc2ac9ef89404bbff1cc8ad5090402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ba2ad8264ef43c7b3539785d4a67533"
I20260928 22:52:58.794988 2775 raft_consensus.cc:3037] T bb399eef861f49f2b4d8a8197647e519 P 7ba2ad8264ef43c7b3539785d4a67533 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:52:58.795725 2909 raft_consensus.cc:2454] T bb399eef861f49f2b4d8a8197647e519 P 68e273fb4ee048eeac9848cb05bfa17b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fc2ac9ef89404bbff1cc8ad5090402 in term 1.
I20260928 22:52:58.795748 2775 raft_consensus.cc:2454] T bb399eef861f49f2b4d8a8197647e519 P 7ba2ad8264ef43c7b3539785d4a67533 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fc2ac9ef89404bbff1cc8ad5090402 in term 1.
I20260928 22:52:58.795904 2577 leader_election.cc:302] T bb399eef861f49f2b4d8a8197647e519 P e3fc2ac9ef89404bbff1cc8ad5090402 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68e273fb4ee048eeac9848cb05bfa17b, e3fc2ac9ef89404bbff1cc8ad5090402; no voters:
I20260928 22:52:58.796020 2980 raft_consensus.cc:2781] T bb399eef861f49f2b4d8a8197647e519 P e3fc2ac9ef89404bbff1cc8ad5090402 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:52:58.796213 2980 raft_consensus.cc:686] T bb399eef861f49f2b4d8a8197647e519 P e3fc2ac9ef89404bbff1cc8ad5090402 [term 1 LEADER]: Becoming Leader. State: Replica: e3fc2ac9ef89404bbff1cc8ad5090402, State: Running, Role: LEADER
I20260928 22:52:58.796341 2980 consensus_queue.cc:237] T bb399eef861f49f2b4d8a8197647e519 P e3fc2ac9ef89404bbff1cc8ad5090402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68e273fb4ee048eeac9848cb05bfa17b" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 43767 } } peers { permanent_uuid: "e3fc2ac9ef89404bbff1cc8ad5090402" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33813 } } peers { permanent_uuid: "7ba2ad8264ef43c7b3539785d4a67533" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38019 } }
I20260928 22:52:58.797302 2496 catalog_manager.cc:5495] T bb399eef861f49f2b4d8a8197647e519 P e3fc2ac9ef89404bbff1cc8ad5090402 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3fc2ac9ef89404bbff1cc8ad5090402 (127.16.121.129). New cstate: current_term: 1 leader_uuid: "e3fc2ac9ef89404bbff1cc8ad5090402" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68e273fb4ee048eeac9848cb05bfa17b" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 43767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3fc2ac9ef89404bbff1cc8ad5090402" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 33813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ba2ad8264ef43c7b3539785d4a67533" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38019 } health_report { overall_health: UNKNOWN } } }
I20260928 22:52:58.907636 2987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:58.904758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47274 (local address 127.16.121.129:33813)
0928 22:52:58.904898 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:58.904900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:58.904911 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:52:58.904957 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:58.904958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:58.904991 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:58.905068 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:52:58.905202 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:58.905703 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:58.906451 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:58.906669 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:58.906730 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:58.906996 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:52:58.907249 (+ 253us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:52:58.907376 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:58.907417 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 22:52:58.907427 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":66,"threads_started":1}
I20260928 22:52:58.909665 2909 raft_consensus.cc:1261] T bb399eef861f49f2b4d8a8197647e519 P 68e273fb4ee048eeac9848cb05bfa17b [term 1 FOLLOWER]: Refusing update from remote peer e3fc2ac9ef89404bbff1cc8ad5090402: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:52:58.909742 2775 raft_consensus.cc:1261] T bb399eef861f49f2b4d8a8197647e519 P 7ba2ad8264ef43c7b3539785d4a67533 [term 1 FOLLOWER]: Refusing update from remote peer e3fc2ac9ef89404bbff1cc8ad5090402: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:52:58.910092 2980 consensus_queue.cc:1035] T bb399eef861f49f2b4d8a8197647e519 P e3fc2ac9ef89404bbff1cc8ad5090402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ba2ad8264ef43c7b3539785d4a67533" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 38019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:52:58.910287 2983 consensus_queue.cc:1035] T bb399eef861f49f2b4d8a8197647e519 P e3fc2ac9ef89404bbff1cc8ad5090402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68e273fb4ee048eeac9848cb05bfa17b" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 43767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:52:58.915480 2495 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:52:58.915591 2495 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:52:58.916395 2495 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4624131371ba4a9ea13f36cd80f92acd: Sending DeleteTablet for 3 replicas of tablet bb399eef861f49f2b4d8a8197647e519
I20260928 22:52:58.916790 2889 tablet_service.cc:1514] Processing DeleteTablet for tablet bb399eef861f49f2b4d8a8197647e519 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:52:58 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:43174
I20260928 22:52:58.916852 2621 tablet_service.cc:1514] Processing DeleteTablet for tablet bb399eef861f49f2b4d8a8197647e519 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:52:58 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:47264
I20260928 22:52:58.916949 2755 tablet_service.cc:1514] Processing DeleteTablet for tablet bb399eef861f49f2b4d8a8197647e519 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:52:58 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:40654
I20260928 22:52:58.917079 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 2558
I20260928 22:52:58.917935 3000 tablet_replica.cc:330] T bb399eef861f49f2b4d8a8197647e519 P 68e273fb4ee048eeac9848cb05bfa17b: stopping tablet replica
I20260928 22:52:58.918066 3000 raft_consensus.cc:2229] T bb399eef861f49f2b4d8a8197647e519 P 68e273fb4ee048eeac9848cb05bfa17b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:52:58.918196 3000 raft_consensus.cc:2258] T bb399eef861f49f2b4d8a8197647e519 P 68e273fb4ee048eeac9848cb05bfa17b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:52:58.918574 3000 ts_tablet_manager.cc:1905] T bb399eef861f49f2b4d8a8197647e519 P 68e273fb4ee048eeac9848cb05bfa17b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:52:58.920038 3002 tablet_replica.cc:330] T bb399eef861f49f2b4d8a8197647e519 P 7ba2ad8264ef43c7b3539785d4a67533: stopping tablet replica
I20260928 22:52:58.920145 3000 ts_tablet_manager.cc:1918] T bb399eef861f49f2b4d8a8197647e519 P 68e273fb4ee048eeac9848cb05bfa17b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:52:58.920171 3002 raft_consensus.cc:2229] T bb399eef861f49f2b4d8a8197647e519 P 7ba2ad8264ef43c7b3539785d4a67533 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:52:58.920243 3000 log.cc:1196] T bb399eef861f49f2b4d8a8197647e519 P 68e273fb4ee048eeac9848cb05bfa17b: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-2/wal/wals/bb399eef861f49f2b4d8a8197647e519
I20260928 22:52:58.920279 3002 raft_consensus.cc:2258] T bb399eef861f49f2b4d8a8197647e519 P 7ba2ad8264ef43c7b3539785d4a67533 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:52:58.920594 3000 ts_tablet_manager.cc:1939] T bb399eef861f49f2b4d8a8197647e519 P 68e273fb4ee048eeac9848cb05bfa17b: Deleting consensus metadata
I20260928 22:52:58.920686 3002 ts_tablet_manager.cc:1905] T bb399eef861f49f2b4d8a8197647e519 P 7ba2ad8264ef43c7b3539785d4a67533: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:52:58.921056 2483 catalog_manager.cc:4841] TS 68e273fb4ee048eeac9848cb05bfa17b (127.16.121.131:43767): tablet bb399eef861f49f2b4d8a8197647e519 (table test-table [id=236df766a0984fe88e1e3a2bf36833a5]) successfully deleted
I20260928 22:52:58.922178 3002 ts_tablet_manager.cc:1918] T bb399eef861f49f2b4d8a8197647e519 P 7ba2ad8264ef43c7b3539785d4a67533: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:52:58.922245 3002 log.cc:1196] T bb399eef861f49f2b4d8a8197647e519 P 7ba2ad8264ef43c7b3539785d4a67533: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790635733329745-16870-0/minicluster-data/ts-1/wal/wals/bb399eef861f49f2b4d8a8197647e519
I20260928 22:52:58.922652 3002 ts_tablet_manager.cc:1939] T bb399eef861f49f2b4d8a8197647e519 P 7ba2ad8264ef43c7b3539785d4a67533: Deleting consensus metadata
I20260928 22:52:58.923112 2481 catalog_manager.cc:4841] TS 7ba2ad8264ef43c7b3539785d4a67533 (127.16.121.130:38019): tablet bb399eef861f49f2b4d8a8197647e519 (table test-table [id=236df766a0984fe88e1e3a2bf36833a5]) successfully deleted
W20260928 22:52:58.924055 2483 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.121.129:33813 (error 108)
I20260928 22:52:58.924378 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 2692
I20260928 22:52:58.927276 3004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:58.924619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:33813 (local address 127.0.0.1:47286)
0928 22:52:58.927155 (+ 2536us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:58.927240 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.129:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2516,"thread_start_us":54,"threads_started":1}
W20260928 22:52:58.927459 2483 catalog_manager.cc:4584] TS e3fc2ac9ef89404bbff1cc8ad5090402 (127.16.121.129:33813): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bb399eef861f49f2b4d8a8197647e519: Network error: Client connection negotiation failed: client connection to 127.16.121.129:33813: connect: Connection refused (error 111)
I20260928 22:52:58.929500 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 2826
I20260928 22:52:58.934872 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 2464
2026-09-28T22:52:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2516 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T22:52:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:52:58Z Disabled control of system clock
I20260928 22:52:58.963178 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:40079
--webserver_interface=127.16.121.190
--webserver_port=0
--builtin_ntp_servers=127.16.121.148:40207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:40079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 22:52:59.045485 3008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:52:59.045727 3008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:52:59.045749 3008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:52:59.046965 3008 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:52:59.047013 3008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:52:59.047026 3008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:52:59.047039 3008 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:52:59.047076 3008 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:52:59.048527 3008 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:40207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:40079
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:40079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.121.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:52:59.048828 3008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:52:59.049209 3008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:52:59.054941 3013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:52:59.054942 3016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:52:59.055126 3014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:52:59.055296 3008 server_base.cc:1034] running on GCE node
I20260928 22:52:59.055567 3008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:52:59.055900 3008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:52:59.057111 3008 hybrid_clock.cc:648] HybridClock initialized: now 1790635979057078 us; error 49 us; skew 500 ppm
I20260928 22:52:59.066068 3008 webserver.cc:460] Webserver started at http://127.16.121.190:39847/ using document root <none> and password file <none>
I20260928 22:52:59.066409 3008 fs_manager.cc:359] Metadata directory not provided
I20260928 22:52:59.066509 3008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:52:59.066675 3008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:52:59.067924 3008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "c4114a5d55c249d880215974ae688a69"
format_stamp: "Formatted at 2026-09-28 22:52:59 on dist-test-slave-56vw"
server_key: "42ef65ad68698a9733cbee192a267590"
server_key_iv: "b2cb09ee0c0bff7506b378e2281f2bf5"
server_key_version: "encryptionkey@0"
I20260928 22:52:59.068320 3008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "c4114a5d55c249d880215974ae688a69"
format_stamp: "Formatted at 2026-09-28 22:52:59 on dist-test-slave-56vw"
server_key: "42ef65ad68698a9733cbee192a267590"
server_key_iv: "b2cb09ee0c0bff7506b378e2281f2bf5"
server_key_version: "encryptionkey@0"
I20260928 22:52:59.069856 3008 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 22:52:59.070710 3022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:59.070974 3008 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:52:59.071133 3008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "c4114a5d55c249d880215974ae688a69"
format_stamp: "Formatted at 2026-09-28 22:52:59 on dist-test-slave-56vw"
server_key: "42ef65ad68698a9733cbee192a267590"
server_key_iv: "b2cb09ee0c0bff7506b378e2281f2bf5"
server_key_version: "encryptionkey@0"
I20260928 22:52:59.071228 3008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:52:59.095230 3008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:52:59.095624 3008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:52:59.095793 3008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:52:59.099916 3008 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.190:40079
I20260928 22:52:59.099956 3074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.190:40079 every 8 connection(s)
I20260928 22:52:59.100337 3008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:52:59.101334 3075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:52:59.104118 3075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4114a5d55c249d880215974ae688a69: Bootstrap starting.
I20260928 22:52:59.104686 3075 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4114a5d55c249d880215974ae688a69: Neither blocks nor log segments found. Creating new log.
I20260928 22:52:59.104933 3075 log.cc:824] T 00000000000000000000000000000000 P c4114a5d55c249d880215974ae688a69: Log is configured to *not* fsync() on all Append() calls
I20260928 22:52:59.105659 3075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4114a5d55c249d880215974ae688a69: No bootstrap required, opened a new log
I20260928 22:52:59.107357 3075 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4114a5d55c249d880215974ae688a69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4114a5d55c249d880215974ae688a69" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 40079 } }
I20260928 22:52:59.107506 3075 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4114a5d55c249d880215974ae688a69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:52:59.107530 3075 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4114a5d55c249d880215974ae688a69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4114a5d55c249d880215974ae688a69, State: Initialized, Role: FOLLOWER
I20260928 22:52:59.107672 3075 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4114a5d55c249d880215974ae688a69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4114a5d55c249d880215974ae688a69" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 40079 } }
I20260928 22:52:59.107748 3075 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4114a5d55c249d880215974ae688a69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:52:59.107771 3075 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4114a5d55c249d880215974ae688a69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:52:59.107800 3075 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4114a5d55c249d880215974ae688a69 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:52:59.108266 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 3008
I20260928 22:52:59.108290 3075 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4114a5d55c249d880215974ae688a69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4114a5d55c249d880215974ae688a69" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 40079 } }
I20260928 22:52:59.108356 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:52:59.108428 3075 leader_election.cc:302] T 00000000000000000000000000000000 P c4114a5d55c249d880215974ae688a69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4114a5d55c249d880215974ae688a69; no voters:
I20260928 22:52:59.108575 16870 external_mini_cluster.cc:1428] Setting key 68c54f874243a0bd19e1c433000c5fba
I20260928 22:52:59.108632 3075 leader_election.cc:288] T 00000000000000000000000000000000 P c4114a5d55c249d880215974ae688a69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:52:59.108706 3078 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4114a5d55c249d880215974ae688a69 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:52:59.108861 3078 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4114a5d55c249d880215974ae688a69 [term 1 LEADER]: Becoming Leader. State: Replica: c4114a5d55c249d880215974ae688a69, State: Running, Role: LEADER
I20260928 22:52:59.108906 3075 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4114a5d55c249d880215974ae688a69 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:52:59.108976 3078 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4114a5d55c249d880215974ae688a69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4114a5d55c249d880215974ae688a69" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 40079 } }
I20260928 22:52:59.109395 3078 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4114a5d55c249d880215974ae688a69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4114a5d55c249d880215974ae688a69. Latest consensus state: current_term: 1 leader_uuid: "c4114a5d55c249d880215974ae688a69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4114a5d55c249d880215974ae688a69" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 40079 } } }
I20260928 22:52:59.109475 3078 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4114a5d55c249d880215974ae688a69 [sys.catalog]: This master's current role is: LEADER
I20260928 22:52:59.109647 3079 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4114a5d55c249d880215974ae688a69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4114a5d55c249d880215974ae688a69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4114a5d55c249d880215974ae688a69" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 40079 } } }
I20260928 22:52:59.109787 3079 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4114a5d55c249d880215974ae688a69 [sys.catalog]: This master's current role is: LEADER
I20260928 22:52:59.110294 3088 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:52:59.110986 3088 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:52:59.112668 3088 catalog_manager.cc:1347] Generated new cluster ID: f8fe449bc6d74dceb86de1267e2393f6
I20260928 22:52:59.112725 3088 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:52:59.114954 3082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:59.109196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50470 (local address 127.16.121.190:40079)
0928 22:52:59.109425 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:59.109434 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:59.109468 (+ 34us) server_negotiation.cc:407] Connection header received
0928 22:52:59.109607 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:59.109615 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:59.109692 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:59.110081 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:52:59.110194 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:59.111583 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:59.112580 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:59.113191 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:59.113280 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:59.114251 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:52:59.114255 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:52:59.114256 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:52:59.114457 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:52:59.114495 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:52:59.114508 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:52:59.114541 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:59.114660 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 22:52:59.114741 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":76,"threads_started":1}
I20260928 22:52:59.126703 3088 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:52:59.127449 3088 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:52:59.141669 3088 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4114a5d55c249d880215974ae688a69: Generated new TSK 0
I20260928 22:52:59.141898 3088 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:52:59.145951 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.129:0
--local_ip_for_outbound_sockets=127.16.121.129
--webserver_interface=127.16.121.129
--webserver_port=0
--tserver_master_addrs=127.16.121.190:40079
--builtin_ntp_servers=127.16.121.148:40207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 22:52:59.230233 3099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:52:59.230497 3099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:52:59.230538 3099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:52:59.231863 3099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:52:59.231937 3099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:52:59.231981 3099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.129
I20260928 22:52:59.233387 3099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:40207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.121.129
--webserver_port=0
--tserver_master_addrs=127.16.121.190:40079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.129
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:52:59.233668 3099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:52:59.233980 3099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 22:52:59.239269 3099 server_base.cc:1034] running on GCE node
W20260928 22:52:59.239264 3104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:52:59.239343 3107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:52:59.239424 3105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:52:59.239725 3099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:52:59.239986 3099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:52:59.241163 3099 hybrid_clock.cc:648] HybridClock initialized: now 1790635979241140 us; error 35 us; skew 500 ppm
I20260928 22:52:59.242657 3099 webserver.cc:460] Webserver started at http://127.16.121.129:33093/ using document root <none> and password file <none>
I20260928 22:52:59.242913 3099 fs_manager.cc:359] Metadata directory not provided
I20260928 22:52:59.242959 3099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:52:59.243108 3099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:52:59.244215 3099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-0/data/instance:
uuid: "097ec637910f4dd69c942bb63af64ca3"
format_stamp: "Formatted at 2026-09-28 22:52:59 on dist-test-slave-56vw"
server_key: "f5fee44b3aa9c12f92db747d7dadc165"
server_key_iv: "712edea477ebcb24abc9e4aa6996e75c"
server_key_version: "encryptionkey@0"
I20260928 22:52:59.244583 3099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance:
uuid: "097ec637910f4dd69c942bb63af64ca3"
format_stamp: "Formatted at 2026-09-28 22:52:59 on dist-test-slave-56vw"
server_key: "f5fee44b3aa9c12f92db747d7dadc165"
server_key_iv: "712edea477ebcb24abc9e4aa6996e75c"
server_key_version: "encryptionkey@0"
I20260928 22:52:59.245996 3099 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:52:59.246794 3113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:59.246974 3099 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:52:59.247105 3099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-0/wal
uuid: "097ec637910f4dd69c942bb63af64ca3"
format_stamp: "Formatted at 2026-09-28 22:52:59 on dist-test-slave-56vw"
server_key: "f5fee44b3aa9c12f92db747d7dadc165"
server_key_iv: "712edea477ebcb24abc9e4aa6996e75c"
server_key_version: "encryptionkey@0"
I20260928 22:52:59.247196 3099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:52:59.265008 3099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:52:59.265378 3099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:52:59.265547 3099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:52:59.265806 3099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:52:59.266162 3099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:52:59.266220 3099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:59.266263 3099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:52:59.266287 3099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:59.272472 3099 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.129:39003
I20260928 22:52:59.272564 3226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.129:39003 every 8 connection(s)
I20260928 22:52:59.272894 3099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
I20260928 22:52:59.278545 3082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:59.273702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:44987 (local address 127.16.121.190:40079)
0928 22:52:59.273801 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:59.273804 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:59.274003 (+ 199us) server_negotiation.cc:407] Connection header received
0928 22:52:59.274559 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:59.274561 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:59.274580 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:59.274645 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:52:59.275462 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:59.275982 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:59.277013 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:59.277250 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:59.277320 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:59.278202 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:52:59.278203 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:52:59.278204 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:52:59.278208 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:52:59.278223 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:52:59.278226 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:52:59.278282 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:59.278443 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 22:52:59.278470 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 22:52:59.278771 3229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:59.273625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:40079 (local address 127.16.121.129:44987)
0928 22:52:59.273842 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:59.273893 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 22:52:59.274456 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:52:59.274725 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:52:59.274729 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:52:59.274738 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:52:59.275357 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:59.275361 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:59.276095 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:59.276096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:59.276899 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:59.276901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:59.277337 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:59.277338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:59.277451 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:59.278053 (+ 602us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:52:59.278054 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:52:59.278084 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:52:59.278107 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:52:59.278329 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:52:59.278335 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:52:59.278338 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:52:59.278420 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 22:52:59.278506 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":70,"threads_started":1}
I20260928 22:52:59.279424 3227 heartbeater.cc:347] Connected to a master server at 127.16.121.190:40079
I20260928 22:52:59.279553 3227 heartbeater.cc:464] Registering TS with master...
I20260928 22:52:59.279771 3227 heartbeater.cc:511] Master 127.16.121.190:40079 requested a full tablet report, sending...
I20260928 22:52:59.280287 3039 ts_manager.cc:194] Registered new tserver with Master: 097ec637910f4dd69c942bb63af64ca3 (127.16.121.129:39003)
I20260928 22:52:59.281018 3039 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.121.129:44987
I20260928 22:52:59.281910 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 3099
I20260928 22:52:59.282032 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance
I20260928 22:52:59.282231 16870 external_mini_cluster.cc:1428] Setting key dfd4ce611083eb05b8f15e575787eb4f
I20260928 22:52:59.283277 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.130:0
--local_ip_for_outbound_sockets=127.16.121.130
--webserver_interface=127.16.121.130
--webserver_port=0
--tserver_master_addrs=127.16.121.190:40079
--builtin_ntp_servers=127.16.121.148:40207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 22:52:59.367280 3230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:52:59.367614 3230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:52:59.367642 3230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:52:59.368898 3230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:52:59.368942 3230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:52:59.368969 3230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.130
I20260928 22:52:59.370371 3230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:40207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.121.130
--webserver_port=0
--tserver_master_addrs=127.16.121.190:40079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.130
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:52:59.370649 3230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:52:59.370970 3230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:52:59.376940 3235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:52:59.376964 3238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:52:59.376992 3230 server_base.cc:1034] running on GCE node
W20260928 22:52:59.376940 3236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:52:59.377399 3230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:52:59.377657 3230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:52:59.378837 3230 hybrid_clock.cc:648] HybridClock initialized: now 1790635979378825 us; error 35 us; skew 500 ppm
I20260928 22:52:59.380255 3230 webserver.cc:460] Webserver started at http://127.16.121.130:44533/ using document root <none> and password file <none>
I20260928 22:52:59.380540 3230 fs_manager.cc:359] Metadata directory not provided
I20260928 22:52:59.380630 3230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:52:59.380781 3230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:52:59.381870 3230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-1/data/instance:
uuid: "d8e79ab1839d43f5b7febb5e82186324"
format_stamp: "Formatted at 2026-09-28 22:52:59 on dist-test-slave-56vw"
server_key: "d6a9f4751df16e614d7f9cfcb6165fb6"
server_key_iv: "9e5688075882dbda379abd5189f41245"
server_key_version: "encryptionkey@0"
I20260928 22:52:59.382241 3230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance:
uuid: "d8e79ab1839d43f5b7febb5e82186324"
format_stamp: "Formatted at 2026-09-28 22:52:59 on dist-test-slave-56vw"
server_key: "d6a9f4751df16e614d7f9cfcb6165fb6"
server_key_iv: "9e5688075882dbda379abd5189f41245"
server_key_version: "encryptionkey@0"
I20260928 22:52:59.383714 3230 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:52:59.384639 3244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:59.384871 3230 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:52:59.384991 3230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-1/wal
uuid: "d8e79ab1839d43f5b7febb5e82186324"
format_stamp: "Formatted at 2026-09-28 22:52:59 on dist-test-slave-56vw"
server_key: "d6a9f4751df16e614d7f9cfcb6165fb6"
server_key_iv: "9e5688075882dbda379abd5189f41245"
server_key_version: "encryptionkey@0"
I20260928 22:52:59.385071 3230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:52:59.395834 3230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:52:59.396162 3230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:52:59.396319 3230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:52:59.396581 3230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:52:59.396925 3230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:52:59.396986 3230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:59.397056 3230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:52:59.397089 3230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:59.402796 3230 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.130:34985
I20260928 22:52:59.402839 3357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.130:34985 every 8 connection(s)
I20260928 22:52:59.403219 3230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
I20260928 22:52:59.408411 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 3230
I20260928 22:52:59.408640 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance
I20260928 22:52:59.408943 16870 external_mini_cluster.cc:1428] Setting key fc83de5f37db444b6755b6d69c3c759c
I20260928 22:52:59.409401 3082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:59.404136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:55093 (local address 127.16.121.190:40079)
0928 22:52:59.404201 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:59.404203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:59.404400 (+ 197us) server_negotiation.cc:407] Connection header received
0928 22:52:59.404929 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:59.404931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:59.404942 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:59.405012 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:52:59.405741 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:59.406255 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:59.407466 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:59.407714 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:59.407782 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:59.408842 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:52:59.408844 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:52:59.408845 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:52:59.408852 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:52:59.408872 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:52:59.408878 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:52:59.408942 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:59.409275 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 22:52:59.409315 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 22:52:59.409466 3360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:59.404084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:40079 (local address 127.16.121.130:55093)
0928 22:52:59.404264 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:59.404306 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 22:52:59.404805 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:52:59.405104 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:52:59.405108 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:52:59.405117 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:52:59.405627 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:59.405631 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:59.406364 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:59.406366 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:59.407364 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:59.407366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:59.407827 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:59.407829 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:59.407965 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:59.408552 (+ 587us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:52:59.408552 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:52:59.408598 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:52:59.408635 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:52:59.408963 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:52:59.408969 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:52:59.408973 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:52:59.409127 (+ 154us) client_negotiation.cc:240] Negotiation successful
0928 22:52:59.409235 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
I20260928 22:52:59.410132 3358 heartbeater.cc:347] Connected to a master server at 127.16.121.190:40079
I20260928 22:52:59.410162 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.131:0
--local_ip_for_outbound_sockets=127.16.121.131
--webserver_interface=127.16.121.131
--webserver_port=0
--tserver_master_addrs=127.16.121.190:40079
--builtin_ntp_servers=127.16.121.148:40207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 22:52:59.410221 3358 heartbeater.cc:464] Registering TS with master...
I20260928 22:52:59.410435 3358 heartbeater.cc:511] Master 127.16.121.190:40079 requested a full tablet report, sending...
I20260928 22:52:59.410840 3039 ts_manager.cc:194] Registered new tserver with Master: d8e79ab1839d43f5b7febb5e82186324 (127.16.121.130:34985)
I20260928 22:52:59.411374 3039 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.121.130:55093
W20260928 22:52:59.496799 3361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:52:59.497068 3361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:52:59.497103 3361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:52:59.498337 3361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:52:59.498404 3361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:52:59.498445 3361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.131
I20260928 22:52:59.499845 3361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:40207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.121.131
--webserver_port=0
--tserver_master_addrs=127.16.121.190:40079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.131
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:52:59.500128 3361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:52:59.500439 3361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:52:59.505873 3367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:52:59.505985 3366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:52:59.505997 3369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:52:59.506279 3361 server_base.cc:1034] running on GCE node
I20260928 22:52:59.506491 3361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:52:59.506690 3361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:52:59.507840 3361 hybrid_clock.cc:648] HybridClock initialized: now 1790635979507826 us; error 31 us; skew 500 ppm
I20260928 22:52:59.509418 3361 webserver.cc:460] Webserver started at http://127.16.121.131:43045/ using document root <none> and password file <none>
I20260928 22:52:59.509692 3361 fs_manager.cc:359] Metadata directory not provided
I20260928 22:52:59.509744 3361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:52:59.509894 3361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:52:59.511173 3361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-2/data/instance:
uuid: "ee4b6d28ba854cb184d9548ea65b0283"
format_stamp: "Formatted at 2026-09-28 22:52:59 on dist-test-slave-56vw"
server_key: "97559ce189ff503e9fdee70893a26cfc"
server_key_iv: "314d7e0ea5409b2a394f461a11aaf650"
server_key_version: "encryptionkey@0"
I20260928 22:52:59.511487 3361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance:
uuid: "ee4b6d28ba854cb184d9548ea65b0283"
format_stamp: "Formatted at 2026-09-28 22:52:59 on dist-test-slave-56vw"
server_key: "97559ce189ff503e9fdee70893a26cfc"
server_key_iv: "314d7e0ea5409b2a394f461a11aaf650"
server_key_version: "encryptionkey@0"
I20260928 22:52:59.512789 3361 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:52:59.513572 3375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:59.513796 3361 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:52:59.513888 3361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-2/wal
uuid: "ee4b6d28ba854cb184d9548ea65b0283"
format_stamp: "Formatted at 2026-09-28 22:52:59 on dist-test-slave-56vw"
server_key: "97559ce189ff503e9fdee70893a26cfc"
server_key_iv: "314d7e0ea5409b2a394f461a11aaf650"
server_key_version: "encryptionkey@0"
I20260928 22:52:59.513967 3361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:52:59.525061 3361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:52:59.525378 3361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:52:59.525529 3361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:52:59.525766 3361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:52:59.526147 3361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:52:59.526181 3361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:59.526206 3361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:52:59.526217 3361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:52:59.531948 3361 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.131:35031
I20260928 22:52:59.531992 3488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.131:35031 every 8 connection(s)
I20260928 22:52:59.532326 3361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
I20260928 22:52:59.536161 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 3361
I20260928 22:52:59.536271 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance
I20260928 22:52:59.536474 16870 external_mini_cluster.cc:1428] Setting key bd7fb6cba3d57a14b5f4cd22b98846d6
I20260928 22:52:59.538098 3082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:59.533114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:42373 (local address 127.16.121.190:40079)
0928 22:52:59.533165 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:59.533167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:59.533383 (+ 216us) server_negotiation.cc:407] Connection header received
0928 22:52:59.533854 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:59.533855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:59.533868 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:59.533933 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:52:59.534627 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:59.535196 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:59.536406 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:59.536729 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:59.536812 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:59.537851 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:52:59.537852 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:52:59.537852 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:52:59.537857 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:52:59.537871 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:52:59.537874 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:52:59.537928 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:59.538012 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 22:52:59.538038 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 22:52:59.538303 3491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:59.533064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:40079 (local address 127.16.121.131:42373)
0928 22:52:59.533250 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:59.533292 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 22:52:59.533767 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:52:59.534029 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:52:59.534033 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:52:59.534042 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:52:59.534531 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:59.534535 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:59.535275 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:59.535276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:59.536253 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:59.536258 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:59.536840 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:59.536841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:59.537029 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:59.537688 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:52:59.537688 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:52:59.537727 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:52:59.537749 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:52:59.537937 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:52:59.537941 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:52:59.537944 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:52:59.537997 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 22:52:59.538103 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":96,"threads_started":1}
I20260928 22:52:59.538789 3489 heartbeater.cc:347] Connected to a master server at 127.16.121.190:40079
I20260928 22:52:59.538894 3489 heartbeater.cc:464] Registering TS with master...
I20260928 22:52:59.539131 3489 heartbeater.cc:511] Master 127.16.121.190:40079 requested a full tablet report, sending...
I20260928 22:52:59.539583 3039 ts_manager.cc:194] Registered new tserver with Master: ee4b6d28ba854cb184d9548ea65b0283 (127.16.121.131:35031)
I20260928 22:52:59.539978 3039 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.121.131:42373
I20260928 22:52:59.548593 16870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:52:59.553295 3082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:59.550094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50484 (local address 127.16.121.190:40079)
0928 22:52:59.550170 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:59.550172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:59.550291 (+ 119us) server_negotiation.cc:407] Connection header received
0928 22:52:59.550352 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:59.550352 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:59.550363 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:59.550421 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:52:59.550631 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:59.551219 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:59.552010 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:59.552214 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:59.552289 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:59.552962 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:52:59.552964 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:52:59.552964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:52:59.552969 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:52:59.552985 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:52:59.552989 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:52:59.553109 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:59.553198 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 22:52:59.553210 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 22:52:59.555156 3039 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50484:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:52:59.556897 3039 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:52:59.563803 3500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:59.559447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41366 (local address 127.16.121.129:39003)
0928 22:52:59.559748 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:59.559751 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:59.559761 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:52:59.559802 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:59.559805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:59.559821 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:59.559916 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:52:59.560241 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:59.561392 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:59.562512 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:59.563302 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:59.563389 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:59.563488 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:59.563679 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 22:52:59.563718 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":102,"threads_started":1}
I20260928 22:52:59.563804 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:59.559200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:39003 (local address 127.0.0.1:41366)
0928 22:52:59.559363 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:59.559409 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 22:52:59.559476 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:52:59.559933 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:52:59.559936 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:52:59.559939 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:52:59.560030 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:59.560032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:59.561497 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:59.561498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:59.562420 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:59.562423 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:59.563409 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:59.563458 (+ 49us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:59.563546 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:59.563565 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 22:52:59.563657 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 22:52:59.563731 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":85,"threads_started":1}
I20260928 22:52:59.563851 3501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:59.559465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58664 (local address 127.16.121.131:35031)
0928 22:52:59.559666 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:59.559671 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:59.560129 (+ 458us) server_negotiation.cc:407] Connection header received
0928 22:52:59.560175 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:59.560177 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:59.560192 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:59.560261 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:52:59.560427 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:59.561163 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:59.562693 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:59.563378 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:59.563445 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:59.563513 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:59.563720 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 22:52:59.563763 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":88,"threads_started":1}
I20260928 22:52:59.564072 3497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:59.559200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.131:35031 (local address 127.0.0.1:58664)
0928 22:52:59.559837 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:59.559849 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:52:59.559904 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:52:59.560327 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:52:59.560328 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:52:59.560329 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:52:59.560364 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:59.560365 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:59.561255 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:59.561256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:59.562601 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:59.562603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:59.563473 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:59.563474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:59.563546 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:59.563565 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 22:52:59.563671 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 22:52:59.564028 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"mutex_wait_us":382,"thread_start_us":107,"threads_started":1}
I20260928 22:52:59.564651 3161 tablet_service.cc:1467] Processing CreateTablet for tablet 1f8426f7299d4d038fabf3497fcd6f4e (DEFAULT_TABLE table=test-table [id=9ec16f24571841d181275f0fc14c3b51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:52:59.565083 3161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f8426f7299d4d038fabf3497fcd6f4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:52:59.564960 3423 tablet_service.cc:1467] Processing CreateTablet for tablet 1f8426f7299d4d038fabf3497fcd6f4e (DEFAULT_TABLE table=test-table [id=9ec16f24571841d181275f0fc14c3b51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:52:59.565313 3423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f8426f7299d4d038fabf3497fcd6f4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:52:59.565534 3499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:59.559200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.130:34985 (local address 127.0.0.1:55128)
0928 22:52:59.559383 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:59.559409 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 22:52:59.559532 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:52:59.560056 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:52:59.560057 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:52:59.560058 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:52:59.560095 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:59.560096 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:59.561444 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:59.561445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:59.562601 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:59.562603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:59.565246 (+ 2643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:59.565247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:59.565315 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:59.565324 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 22:52:59.565407 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 22:52:59.565453 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":24,"thread_start_us":65,"threads_started":1}
I20260928 22:52:59.565786 3502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:59.559647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55128 (local address 127.16.121.130:34985)
0928 22:52:59.559823 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:59.559827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:59.559835 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:52:59.559866 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:59.559868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:59.559900 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:59.559986 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:52:59.560160 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:59.561357 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:59.563550 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:59.565044 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:59.565137 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:59.565227 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:59.565458 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 22:52:59.565503 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":96,"threads_started":1}
I20260928 22:52:59.566677 3292 tablet_service.cc:1467] Processing CreateTablet for tablet 1f8426f7299d4d038fabf3497fcd6f4e (DEFAULT_TABLE table=test-table [id=9ec16f24571841d181275f0fc14c3b51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:52:59.567003 3292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f8426f7299d4d038fabf3497fcd6f4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:52:59.570194 3503 tablet_bootstrap.cc:492] T 1f8426f7299d4d038fabf3497fcd6f4e P ee4b6d28ba854cb184d9548ea65b0283: Bootstrap starting.
I20260928 22:52:59.570802 3505 tablet_bootstrap.cc:492] T 1f8426f7299d4d038fabf3497fcd6f4e P d8e79ab1839d43f5b7febb5e82186324: Bootstrap starting.
I20260928 22:52:59.570958 3503 tablet_bootstrap.cc:654] T 1f8426f7299d4d038fabf3497fcd6f4e P ee4b6d28ba854cb184d9548ea65b0283: Neither blocks nor log segments found. Creating new log.
I20260928 22:52:59.571148 3504 tablet_bootstrap.cc:492] T 1f8426f7299d4d038fabf3497fcd6f4e P 097ec637910f4dd69c942bb63af64ca3: Bootstrap starting.
I20260928 22:52:59.571523 3503 log.cc:824] T 1f8426f7299d4d038fabf3497fcd6f4e P ee4b6d28ba854cb184d9548ea65b0283: Log is configured to *not* fsync() on all Append() calls
I20260928 22:52:59.571565 3505 tablet_bootstrap.cc:654] T 1f8426f7299d4d038fabf3497fcd6f4e P d8e79ab1839d43f5b7febb5e82186324: Neither blocks nor log segments found. Creating new log.
I20260928 22:52:59.571883 3505 log.cc:824] T 1f8426f7299d4d038fabf3497fcd6f4e P d8e79ab1839d43f5b7febb5e82186324: Log is configured to *not* fsync() on all Append() calls
I20260928 22:52:59.572010 3504 tablet_bootstrap.cc:654] T 1f8426f7299d4d038fabf3497fcd6f4e P 097ec637910f4dd69c942bb63af64ca3: Neither blocks nor log segments found. Creating new log.
I20260928 22:52:59.572286 3503 tablet_bootstrap.cc:492] T 1f8426f7299d4d038fabf3497fcd6f4e P ee4b6d28ba854cb184d9548ea65b0283: No bootstrap required, opened a new log
I20260928 22:52:59.572299 3504 log.cc:824] T 1f8426f7299d4d038fabf3497fcd6f4e P 097ec637910f4dd69c942bb63af64ca3: Log is configured to *not* fsync() on all Append() calls
I20260928 22:52:59.572371 3503 ts_tablet_manager.cc:1397] T 1f8426f7299d4d038fabf3497fcd6f4e P ee4b6d28ba854cb184d9548ea65b0283: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:52:59.573002 3504 tablet_bootstrap.cc:492] T 1f8426f7299d4d038fabf3497fcd6f4e P 097ec637910f4dd69c942bb63af64ca3: No bootstrap required, opened a new log
I20260928 22:52:59.573002 3505 tablet_bootstrap.cc:492] T 1f8426f7299d4d038fabf3497fcd6f4e P d8e79ab1839d43f5b7febb5e82186324: No bootstrap required, opened a new log
I20260928 22:52:59.573072 3504 ts_tablet_manager.cc:1397] T 1f8426f7299d4d038fabf3497fcd6f4e P 097ec637910f4dd69c942bb63af64ca3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 22:52:59.573073 3505 ts_tablet_manager.cc:1397] T 1f8426f7299d4d038fabf3497fcd6f4e P d8e79ab1839d43f5b7febb5e82186324: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 22:52:59.574569 3503 raft_consensus.cc:348] T 1f8426f7299d4d038fabf3497fcd6f4e P ee4b6d28ba854cb184d9548ea65b0283 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4b6d28ba854cb184d9548ea65b0283" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 35031 } } peers { permanent_uuid: "d8e79ab1839d43f5b7febb5e82186324" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 34985 } } peers { permanent_uuid: "097ec637910f4dd69c942bb63af64ca3" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 39003 } }
I20260928 22:52:59.574730 3503 raft_consensus.cc:374] T 1f8426f7299d4d038fabf3497fcd6f4e P ee4b6d28ba854cb184d9548ea65b0283 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:52:59.574774 3503 raft_consensus.cc:729] T 1f8426f7299d4d038fabf3497fcd6f4e P ee4b6d28ba854cb184d9548ea65b0283 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee4b6d28ba854cb184d9548ea65b0283, State: Initialized, Role: FOLLOWER
I20260928 22:52:59.574882 3503 consensus_queue.cc:260] T 1f8426f7299d4d038fabf3497fcd6f4e P ee4b6d28ba854cb184d9548ea65b0283 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4b6d28ba854cb184d9548ea65b0283" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 35031 } } peers { permanent_uuid: "d8e79ab1839d43f5b7febb5e82186324" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 34985 } } peers { permanent_uuid: "097ec637910f4dd69c942bb63af64ca3" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 39003 } }
I20260928 22:52:59.575170 3503 ts_tablet_manager.cc:1428] T 1f8426f7299d4d038fabf3497fcd6f4e P ee4b6d28ba854cb184d9548ea65b0283: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:52:59.575327 3489 heartbeater.cc:503] Master 127.16.121.190:40079 was elected leader, sending a full tablet report...
I20260928 22:52:59.575766 3504 raft_consensus.cc:348] T 1f8426f7299d4d038fabf3497fcd6f4e P 097ec637910f4dd69c942bb63af64ca3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4b6d28ba854cb184d9548ea65b0283" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 35031 } } peers { permanent_uuid: "d8e79ab1839d43f5b7febb5e82186324" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 34985 } } peers { permanent_uuid: "097ec637910f4dd69c942bb63af64ca3" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 39003 } }
I20260928 22:52:59.575834 3505 raft_consensus.cc:348] T 1f8426f7299d4d038fabf3497fcd6f4e P d8e79ab1839d43f5b7febb5e82186324 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4b6d28ba854cb184d9548ea65b0283" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 35031 } } peers { permanent_uuid: "d8e79ab1839d43f5b7febb5e82186324" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 34985 } } peers { permanent_uuid: "097ec637910f4dd69c942bb63af64ca3" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 39003 } }
I20260928 22:52:59.575932 3504 raft_consensus.cc:374] T 1f8426f7299d4d038fabf3497fcd6f4e P 097ec637910f4dd69c942bb63af64ca3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:52:59.575963 3505 raft_consensus.cc:374] T 1f8426f7299d4d038fabf3497fcd6f4e P d8e79ab1839d43f5b7febb5e82186324 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:52:59.575966 3504 raft_consensus.cc:729] T 1f8426f7299d4d038fabf3497fcd6f4e P 097ec637910f4dd69c942bb63af64ca3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 097ec637910f4dd69c942bb63af64ca3, State: Initialized, Role: FOLLOWER
I20260928 22:52:59.575994 3505 raft_consensus.cc:729] T 1f8426f7299d4d038fabf3497fcd6f4e P d8e79ab1839d43f5b7febb5e82186324 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8e79ab1839d43f5b7febb5e82186324, State: Initialized, Role: FOLLOWER
I20260928 22:52:59.576116 3505 consensus_queue.cc:260] T 1f8426f7299d4d038fabf3497fcd6f4e P d8e79ab1839d43f5b7febb5e82186324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4b6d28ba854cb184d9548ea65b0283" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 35031 } } peers { permanent_uuid: "d8e79ab1839d43f5b7febb5e82186324" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 34985 } } peers { permanent_uuid: "097ec637910f4dd69c942bb63af64ca3" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 39003 } }
I20260928 22:52:59.576144 3504 consensus_queue.cc:260] T 1f8426f7299d4d038fabf3497fcd6f4e P 097ec637910f4dd69c942bb63af64ca3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4b6d28ba854cb184d9548ea65b0283" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 35031 } } peers { permanent_uuid: "d8e79ab1839d43f5b7febb5e82186324" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 34985 } } peers { permanent_uuid: "097ec637910f4dd69c942bb63af64ca3" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 39003 } }
I20260928 22:52:59.576382 3505 ts_tablet_manager.cc:1428] T 1f8426f7299d4d038fabf3497fcd6f4e P d8e79ab1839d43f5b7febb5e82186324: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 22:52:59.576382 3504 ts_tablet_manager.cc:1428] T 1f8426f7299d4d038fabf3497fcd6f4e P 097ec637910f4dd69c942bb63af64ca3: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 22:52:59.576502 3358 heartbeater.cc:503] Master 127.16.121.190:40079 was elected leader, sending a full tablet report...
I20260928 22:52:59.576502 3227 heartbeater.cc:503] Master 127.16.121.190:40079 was elected leader, sending a full tablet report...
I20260928 22:52:59.576382 3511 raft_consensus.cc:482] T 1f8426f7299d4d038fabf3497fcd6f4e P 097ec637910f4dd69c942bb63af64ca3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:52:59.576758 3511 raft_consensus.cc:504] T 1f8426f7299d4d038fabf3497fcd6f4e P 097ec637910f4dd69c942bb63af64ca3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4b6d28ba854cb184d9548ea65b0283" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 35031 } } peers { permanent_uuid: "d8e79ab1839d43f5b7febb5e82186324" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 34985 } } peers { permanent_uuid: "097ec637910f4dd69c942bb63af64ca3" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 39003 } }
I20260928 22:52:59.577075 3511 leader_election.cc:288] T 1f8426f7299d4d038fabf3497fcd6f4e P 097ec637910f4dd69c942bb63af64ca3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee4b6d28ba854cb184d9548ea65b0283 (127.16.121.131:35031), d8e79ab1839d43f5b7febb5e82186324 (127.16.121.130:34985)
I20260928 22:52:59.580044 3229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:59.577224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.131:35031 (local address 127.16.121.129:35227)
0928 22:52:59.577260 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:59.577272 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:52:59.577315 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:52:59.577555 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:52:59.577556 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:52:59.577558 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:52:59.577630 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:59.577632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:59.578344 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:59.578345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:59.579255 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:59.579258 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:59.579805 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:59.579806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:59.579872 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:59.579876 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:52:59.579935 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 22:52:59.579978 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18,"mutex_wait_us":37}
I20260928 22:52:59.580055 3501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:59.577378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:35227 (local address 127.16.121.131:35031)
0928 22:52:59.577420 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:59.577422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:59.577430 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:52:59.577453 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:59.577454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:59.577463 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:59.577509 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:52:59.577710 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:59.578265 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:59.579336 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:59.579730 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:59.579784 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:59.579791 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:59.579949 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 22:52:59.579986 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 22:52:59.580197 3513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:59.577239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.130:34985 (local address 127.16.121.129:55445)
0928 22:52:59.577517 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:59.577527 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:52:59.577606 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:52:59.577780 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:52:59.577781 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:52:59.577782 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:52:59.577851 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:59.577853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:59.578490 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:59.578491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:59.579384 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:52:59.579387 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:52:59.579955 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:52:59.579956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:52:59.580022 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:59.580026 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:52:59.580085 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 22:52:59.580119 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":85,"threads_started":1}
I20260928 22:52:59.580219 3502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:59.577350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:55445 (local address 127.16.121.130:34985)
0928 22:52:59.577389 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:59.577391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:59.577576 (+ 185us) server_negotiation.cc:407] Connection header received
0928 22:52:59.577680 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:59.577680 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:59.577689 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:59.577717 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:52:59.577906 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:59.578410 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:59.579466 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:59.579844 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:59.579924 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:59.579932 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:59.580104 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 22:52:59.580147 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:52:59.580397 3443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f8426f7299d4d038fabf3497fcd6f4e" candidate_uuid: "097ec637910f4dd69c942bb63af64ca3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee4b6d28ba854cb184d9548ea65b0283" is_pre_election: true
I20260928 22:52:59.580529 3443 raft_consensus.cc:2454] T 1f8426f7299d4d038fabf3497fcd6f4e P ee4b6d28ba854cb184d9548ea65b0283 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 097ec637910f4dd69c942bb63af64ca3 in term 0.
I20260928 22:52:59.580506 3312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f8426f7299d4d038fabf3497fcd6f4e" candidate_uuid: "097ec637910f4dd69c942bb63af64ca3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8e79ab1839d43f5b7febb5e82186324" is_pre_election: true
I20260928 22:52:59.580608 3312 raft_consensus.cc:2454] T 1f8426f7299d4d038fabf3497fcd6f4e P d8e79ab1839d43f5b7febb5e82186324 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 097ec637910f4dd69c942bb63af64ca3 in term 0.
I20260928 22:52:59.580744 3116 leader_election.cc:302] T 1f8426f7299d4d038fabf3497fcd6f4e P 097ec637910f4dd69c942bb63af64ca3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 097ec637910f4dd69c942bb63af64ca3, ee4b6d28ba854cb184d9548ea65b0283; no voters:
I20260928 22:52:59.580890 3510 raft_consensus.cc:2781] T 1f8426f7299d4d038fabf3497fcd6f4e P 097ec637910f4dd69c942bb63af64ca3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:52:59.580978 3510 raft_consensus.cc:482] T 1f8426f7299d4d038fabf3497fcd6f4e P 097ec637910f4dd69c942bb63af64ca3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:52:59.581007 3510 raft_consensus.cc:3037] T 1f8426f7299d4d038fabf3497fcd6f4e P 097ec637910f4dd69c942bb63af64ca3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:52:59.581771 3510 raft_consensus.cc:504] T 1f8426f7299d4d038fabf3497fcd6f4e P 097ec637910f4dd69c942bb63af64ca3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4b6d28ba854cb184d9548ea65b0283" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 35031 } } peers { permanent_uuid: "d8e79ab1839d43f5b7febb5e82186324" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 34985 } } peers { permanent_uuid: "097ec637910f4dd69c942bb63af64ca3" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 39003 } }
I20260928 22:52:59.581909 3510 leader_election.cc:288] T 1f8426f7299d4d038fabf3497fcd6f4e P 097ec637910f4dd69c942bb63af64ca3 [CANDIDATE]: Term 1 election: Requested vote from peers ee4b6d28ba854cb184d9548ea65b0283 (127.16.121.131:35031), d8e79ab1839d43f5b7febb5e82186324 (127.16.121.130:34985)
I20260928 22:52:59.582082 3312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f8426f7299d4d038fabf3497fcd6f4e" candidate_uuid: "097ec637910f4dd69c942bb63af64ca3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8e79ab1839d43f5b7febb5e82186324"
I20260928 22:52:59.582134 3443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f8426f7299d4d038fabf3497fcd6f4e" candidate_uuid: "097ec637910f4dd69c942bb63af64ca3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee4b6d28ba854cb184d9548ea65b0283"
I20260928 22:52:59.582163 3312 raft_consensus.cc:3037] T 1f8426f7299d4d038fabf3497fcd6f4e P d8e79ab1839d43f5b7febb5e82186324 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:52:59.582194 3443 raft_consensus.cc:3037] T 1f8426f7299d4d038fabf3497fcd6f4e P ee4b6d28ba854cb184d9548ea65b0283 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:52:59.583021 3312 raft_consensus.cc:2454] T 1f8426f7299d4d038fabf3497fcd6f4e P d8e79ab1839d43f5b7febb5e82186324 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 097ec637910f4dd69c942bb63af64ca3 in term 1.
I20260928 22:52:59.583021 3443 raft_consensus.cc:2454] T 1f8426f7299d4d038fabf3497fcd6f4e P ee4b6d28ba854cb184d9548ea65b0283 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 097ec637910f4dd69c942bb63af64ca3 in term 1.
I20260928 22:52:59.583210 3116 leader_election.cc:302] T 1f8426f7299d4d038fabf3497fcd6f4e P 097ec637910f4dd69c942bb63af64ca3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 097ec637910f4dd69c942bb63af64ca3, ee4b6d28ba854cb184d9548ea65b0283; no voters:
I20260928 22:52:59.583313 3510 raft_consensus.cc:2781] T 1f8426f7299d4d038fabf3497fcd6f4e P 097ec637910f4dd69c942bb63af64ca3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:52:59.583362 3510 raft_consensus.cc:686] T 1f8426f7299d4d038fabf3497fcd6f4e P 097ec637910f4dd69c942bb63af64ca3 [term 1 LEADER]: Becoming Leader. State: Replica: 097ec637910f4dd69c942bb63af64ca3, State: Running, Role: LEADER
I20260928 22:52:59.583433 3510 consensus_queue.cc:237] T 1f8426f7299d4d038fabf3497fcd6f4e P 097ec637910f4dd69c942bb63af64ca3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4b6d28ba854cb184d9548ea65b0283" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 35031 } } peers { permanent_uuid: "d8e79ab1839d43f5b7febb5e82186324" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 34985 } } peers { permanent_uuid: "097ec637910f4dd69c942bb63af64ca3" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 39003 } }
I20260928 22:52:59.584275 3039 catalog_manager.cc:5495] T 1f8426f7299d4d038fabf3497fcd6f4e P 097ec637910f4dd69c942bb63af64ca3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 097ec637910f4dd69c942bb63af64ca3 (127.16.121.129). New cstate: current_term: 1 leader_uuid: "097ec637910f4dd69c942bb63af64ca3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee4b6d28ba854cb184d9548ea65b0283" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 35031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8e79ab1839d43f5b7febb5e82186324" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 34985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "097ec637910f4dd69c942bb63af64ca3" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 39003 } health_report { overall_health: HEALTHY } } }
I20260928 22:52:59.592258 3500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:59.589798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41368 (local address 127.16.121.129:39003)
0928 22:52:59.589840 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 22:52:59.589842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:52:59.589851 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:52:59.589897 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:52:59.589898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:52:59.589915 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:52:59.589987 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:52:59.590108 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:59.590617 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:59.591420 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:52:59.591643 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:52:59.591715 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:52:59.591901 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:52:59.591994 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:52:59.592075 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 22:52:59.592185 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 22:52:59.592195 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:52:59.594285 3312 raft_consensus.cc:1261] T 1f8426f7299d4d038fabf3497fcd6f4e P d8e79ab1839d43f5b7febb5e82186324 [term 1 FOLLOWER]: Refusing update from remote peer 097ec637910f4dd69c942bb63af64ca3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:52:59.594333 3443 raft_consensus.cc:1261] T 1f8426f7299d4d038fabf3497fcd6f4e P ee4b6d28ba854cb184d9548ea65b0283 [term 1 FOLLOWER]: Refusing update from remote peer 097ec637910f4dd69c942bb63af64ca3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:52:59.594720 3511 consensus_queue.cc:1035] T 1f8426f7299d4d038fabf3497fcd6f4e P 097ec637910f4dd69c942bb63af64ca3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8e79ab1839d43f5b7febb5e82186324" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 34985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:52:59.594859 3510 consensus_queue.cc:1035] T 1f8426f7299d4d038fabf3497fcd6f4e P 097ec637910f4dd69c942bb63af64ca3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4b6d28ba854cb184d9548ea65b0283" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 35031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:52:59.597926 3516 mvcc.cc:205] Tried to move back new op lower bound from 7334444972414844928 to 7334444972374417408. Current Snapshot: MvccSnapshot[applied={T|T < 7334444972414844928}]
I20260928 22:52:59.600845 3039 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:52:59.600960 3039 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:52:59.601749 3039 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c4114a5d55c249d880215974ae688a69: Sending DeleteTablet for 3 replicas of tablet 1f8426f7299d4d038fabf3497fcd6f4e
I20260928 22:52:59.602099 3423 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f8426f7299d4d038fabf3497fcd6f4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:52:59 UTC) from {username='slave'} at 127.0.0.1:58664
I20260928 22:52:59.602101 3161 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f8426f7299d4d038fabf3497fcd6f4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:52:59 UTC) from {username='slave'} at 127.0.0.1:41366
I20260928 22:52:59.602105 3292 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f8426f7299d4d038fabf3497fcd6f4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:52:59 UTC) from {username='slave'} at 127.0.0.1:55128
I20260928 22:52:59.602360 3528 tablet_replica.cc:330] T 1f8426f7299d4d038fabf3497fcd6f4e P d8e79ab1839d43f5b7febb5e82186324: stopping tablet replica
I20260928 22:52:59.602455 3527 tablet_replica.cc:330] T 1f8426f7299d4d038fabf3497fcd6f4e P 097ec637910f4dd69c942bb63af64ca3: stopping tablet replica
I20260928 22:52:59.602558 3527 raft_consensus.cc:2229] T 1f8426f7299d4d038fabf3497fcd6f4e P 097ec637910f4dd69c942bb63af64ca3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 22:52:59.602560 3528 raft_consensus.cc:2229] T 1f8426f7299d4d038fabf3497fcd6f4e P d8e79ab1839d43f5b7febb5e82186324 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:52:59.602646 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 3099
I20260928 22:52:59.603353 3528 raft_consensus.cc:2258] T 1f8426f7299d4d038fabf3497fcd6f4e P d8e79ab1839d43f5b7febb5e82186324 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:52:59.603998 3528 ts_tablet_manager.cc:1905] T 1f8426f7299d4d038fabf3497fcd6f4e P d8e79ab1839d43f5b7febb5e82186324: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:52:59.604673 3526 tablet_replica.cc:330] T 1f8426f7299d4d038fabf3497fcd6f4e P ee4b6d28ba854cb184d9548ea65b0283: stopping tablet replica
I20260928 22:52:59.604796 3526 raft_consensus.cc:2229] T 1f8426f7299d4d038fabf3497fcd6f4e P ee4b6d28ba854cb184d9548ea65b0283 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:52:59.604903 3526 raft_consensus.cc:2258] T 1f8426f7299d4d038fabf3497fcd6f4e P ee4b6d28ba854cb184d9548ea65b0283 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:52:59.605444 3526 ts_tablet_manager.cc:1905] T 1f8426f7299d4d038fabf3497fcd6f4e P ee4b6d28ba854cb184d9548ea65b0283: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:52:59.606154 3528 ts_tablet_manager.cc:1918] T 1f8426f7299d4d038fabf3497fcd6f4e P d8e79ab1839d43f5b7febb5e82186324: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:52:59.606256 3528 log.cc:1196] T 1f8426f7299d4d038fabf3497fcd6f4e P d8e79ab1839d43f5b7febb5e82186324: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-1/wal/wals/1f8426f7299d4d038fabf3497fcd6f4e
I20260928 22:52:59.606527 3528 ts_tablet_manager.cc:1939] T 1f8426f7299d4d038fabf3497fcd6f4e P d8e79ab1839d43f5b7febb5e82186324: Deleting consensus metadata
I20260928 22:52:59.606945 3526 ts_tablet_manager.cc:1918] T 1f8426f7299d4d038fabf3497fcd6f4e P ee4b6d28ba854cb184d9548ea65b0283: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:52:59.606952 3026 catalog_manager.cc:4841] TS d8e79ab1839d43f5b7febb5e82186324 (127.16.121.130:34985): tablet 1f8426f7299d4d038fabf3497fcd6f4e (table test-table [id=9ec16f24571841d181275f0fc14c3b51]) successfully deleted
I20260928 22:52:59.607035 3526 log.cc:1196] T 1f8426f7299d4d038fabf3497fcd6f4e P ee4b6d28ba854cb184d9548ea65b0283: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790635733329745-16870-0/minicluster-data/ts-2/wal/wals/1f8426f7299d4d038fabf3497fcd6f4e
I20260928 22:52:59.607295 3526 ts_tablet_manager.cc:1939] T 1f8426f7299d4d038fabf3497fcd6f4e P ee4b6d28ba854cb184d9548ea65b0283: Deleting consensus metadata
I20260928 22:52:59.608676 3025 catalog_manager.cc:4841] TS ee4b6d28ba854cb184d9548ea65b0283 (127.16.121.131:35031): tablet 1f8426f7299d4d038fabf3497fcd6f4e (table test-table [id=9ec16f24571841d181275f0fc14c3b51]) successfully deleted
W20260928 22:52:59.609282 3024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.121.129:39003 (error 108)
I20260928 22:52:59.609490 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 3230
I20260928 22:52:59.611493 3499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:52:59.611327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:39003 (local address 127.0.0.1:41382)
0928 22:52:59.611392 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 22:52:59.611455 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.129:39003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 22:52:59.611624 3024 catalog_manager.cc:4584] TS 097ec637910f4dd69c942bb63af64ca3 (127.16.121.129:39003): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1f8426f7299d4d038fabf3497fcd6f4e: Network error: Client connection negotiation failed: client connection to 127.16.121.129:39003: connect: Connection refused (error 111)
I20260928 22:52:59.614789 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 3361
I20260928 22:52:59.619822 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 3008
2026-09-28T22:52:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (684 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:52:59 dist-test-slave-56vw krb5kdc[3532](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:52:59 dist-test-slave-56vw krb5kdc[3532](info): set up 2 sockets
Sep 28 22:52:59 dist-test-slave-56vw krb5kdc[3532](info): commencing operation
krb5kdc: starting...
W20260928 22:53:00.674847 16870 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Sep 28 22:53:00 dist-test-slave-56vw krb5kdc[3532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:53:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:53:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.121.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
I20260928 22:53:00.811038 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:40667
--webserver_interface=127.16.121.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:39277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:40667
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:00.896090 3545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:00.896306 3545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:00.896329 3545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:00.897521 3545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:53:00.897572 3545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:00.897588 3545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:00.897600 3545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:53:00.897612 3545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:53:00.899201 3545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:39277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:40667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:40667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.121.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:00.899464 3545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:00.899725 3545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:00.904888 3553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:00.904888 3551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:00.904903 3550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:00.905297 3545 server_base.cc:1034] running on GCE node
I20260928 22:53:00.905530 3545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:00.905859 3545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:00.907008 3545 hybrid_clock.cc:648] HybridClock initialized: now 1790635980906984 us; error 42 us; skew 500 ppm
Sep 28 22:53:00 dist-test-slave-56vw krb5kdc[3532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635980, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:00.909912 3545 init.cc:377] Logged in from keytab as kudu/127.16.121.190@KRBTEST.COM (short username kudu)
I20260928 22:53:00.910550 3545 webserver.cc:460] Webserver started at http://127.16.121.190:40999/ using document root <none> and password file <none>
I20260928 22:53:00.910792 3545 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:00.910863 3545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:00.911019 3545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:00.912149 3545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "1fc3de5a78d24b699d99fb31468fbae5"
format_stamp: "Formatted at 2026-09-28 22:53:00 on dist-test-slave-56vw"
server_key: "b7f766f8683ec7af243e7e8faf6aebbf"
server_key_iv: "b05f7a64e04b578064ed4667a81f6c2a"
server_key_version: "encryptionkey@0"
I20260928 22:53:00.912531 3545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "1fc3de5a78d24b699d99fb31468fbae5"
format_stamp: "Formatted at 2026-09-28 22:53:00 on dist-test-slave-56vw"
server_key: "b7f766f8683ec7af243e7e8faf6aebbf"
server_key_iv: "b05f7a64e04b578064ed4667a81f6c2a"
server_key_version: "encryptionkey@0"
I20260928 22:53:00.913913 3545 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:53:00.914693 3560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:00.914899 3545 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 22:53:00.915001 3545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "1fc3de5a78d24b699d99fb31468fbae5"
format_stamp: "Formatted at 2026-09-28 22:53:00 on dist-test-slave-56vw"
server_key: "b7f766f8683ec7af243e7e8faf6aebbf"
server_key_iv: "b05f7a64e04b578064ed4667a81f6c2a"
server_key_version: "encryptionkey@0"
I20260928 22:53:00.915163 3545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:00.925868 3545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:00.928411 3545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:00.928586 3545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:00.933103 3545 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.190:40667
I20260928 22:53:00.933138 3612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.190:40667 every 8 connection(s)
I20260928 22:53:00.933534 3545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:53:00.934477 3613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:00.936404 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 3545
I20260928 22:53:00.936523 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:53:00.936714 16870 external_mini_cluster.cc:1428] Setting key 9ddd4cd24214ed850e1454a58540c195
I20260928 22:53:00.937371 3613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fc3de5a78d24b699d99fb31468fbae5: Bootstrap starting.
I20260928 22:53:00.937981 3613 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fc3de5a78d24b699d99fb31468fbae5: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:00.938277 3613 log.cc:824] T 00000000000000000000000000000000 P 1fc3de5a78d24b699d99fb31468fbae5: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:00.939194 3613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fc3de5a78d24b699d99fb31468fbae5: No bootstrap required, opened a new log
I20260928 22:53:00.941262 3613 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fc3de5a78d24b699d99fb31468fbae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fc3de5a78d24b699d99fb31468fbae5" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 40667 } }
I20260928 22:53:00.941433 3613 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1fc3de5a78d24b699d99fb31468fbae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:00.941470 3613 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fc3de5a78d24b699d99fb31468fbae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fc3de5a78d24b699d99fb31468fbae5, State: Initialized, Role: FOLLOWER
I20260928 22:53:00.941638 3613 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fc3de5a78d24b699d99fb31468fbae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fc3de5a78d24b699d99fb31468fbae5" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 40667 } }
I20260928 22:53:00.941731 3613 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1fc3de5a78d24b699d99fb31468fbae5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:53:00.941769 3613 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fc3de5a78d24b699d99fb31468fbae5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:53:00.941816 3613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fc3de5a78d24b699d99fb31468fbae5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:00.942420 3613 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fc3de5a78d24b699d99fb31468fbae5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fc3de5a78d24b699d99fb31468fbae5" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 40667 } }
I20260928 22:53:00.942545 3613 leader_election.cc:302] T 00000000000000000000000000000000 P 1fc3de5a78d24b699d99fb31468fbae5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fc3de5a78d24b699d99fb31468fbae5; no voters:
I20260928 22:53:00.942732 3613 leader_election.cc:288] T 00000000000000000000000000000000 P 1fc3de5a78d24b699d99fb31468fbae5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:53:00.942797 3618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fc3de5a78d24b699d99fb31468fbae5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:53:00.942946 3618 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fc3de5a78d24b699d99fb31468fbae5 [term 1 LEADER]: Becoming Leader. State: Replica: 1fc3de5a78d24b699d99fb31468fbae5, State: Running, Role: LEADER
I20260928 22:53:00.943023 3613 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fc3de5a78d24b699d99fb31468fbae5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:53:00.943073 3618 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fc3de5a78d24b699d99fb31468fbae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fc3de5a78d24b699d99fb31468fbae5" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 40667 } }
I20260928 22:53:00.943423 3619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fc3de5a78d24b699d99fb31468fbae5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1fc3de5a78d24b699d99fb31468fbae5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fc3de5a78d24b699d99fb31468fbae5" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 40667 } } }
I20260928 22:53:00.943516 3619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fc3de5a78d24b699d99fb31468fbae5 [sys.catalog]: This master's current role is: LEADER
Sep 28 22:53:00 dist-test-slave-56vw krb5kdc[3532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:53:00.943411 3618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fc3de5a78d24b699d99fb31468fbae5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fc3de5a78d24b699d99fb31468fbae5. Latest consensus state: current_term: 1 leader_uuid: "1fc3de5a78d24b699d99fb31468fbae5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fc3de5a78d24b699d99fb31468fbae5" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 40667 } } }
I20260928 22:53:00.943712 3618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fc3de5a78d24b699d99fb31468fbae5 [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:00.943835 3624 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:53:00.944345 3624 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:53:00.945886 3624 catalog_manager.cc:1347] Generated new cluster ID: af230e3c65d84496908d5c2138248ea9
I20260928 22:53:00.945932 3624 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:53:00.950644 3616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:00.937304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54172 (local address 127.16.121.190:40667)
0928 22:53:00.937597 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:00.937606 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:00.937634 (+ 28us) server_negotiation.cc:407] Connection header received
0928 22:53:00.938070 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:00.938103 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:00.938181 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:00.938353 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:00.939157 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:00.940163 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:00.940944 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:00.941390 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:00.941513 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:00.944259 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:00.944264 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:00.944265 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:00.944277 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:00.948911 (+ 4634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:00.949347 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:00.949351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:00.949351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:00.949471 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:00.949681 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:00.949681 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:00.949682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:00.950011 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:00.950188 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:00.950276 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 22:53:00.950427 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":86,"threads_started":1}
I20260928 22:53:00.956127 3624 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:53:00.956730 3624 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:53:00.959517 3624 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1fc3de5a78d24b699d99fb31468fbae5: Generated new TSK 0
I20260928 22:53:00.959728 3624 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.121.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab.
Entry for principal kudu/127.16.121.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab.
I20260928 22:53:00.999405 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.129:0
--local_ip_for_outbound_sockets=127.16.121.129
--webserver_interface=127.16.121.129
--webserver_port=0
--tserver_master_addrs=127.16.121.190:40667
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab
--principal=kudu/127.16.121.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:39277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:01.103605 3639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:01.103845 3639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:01.103888 3639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:01.105106 3639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:01.105178 3639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:01.105226 3639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.129
I20260928 22:53:01.106765 3639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:39277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.129.keytab
--principal=kudu/127.16.121.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.121.129
--webserver_port=0
--tserver_master_addrs=127.16.121.190:40667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.129
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:01.107120 3639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:01.107365 3639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:01.113914 3645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:01.114039 3639 server_base.cc:1034] running on GCE node
W20260928 22:53:01.113919 3647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:01.113919 3644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:01.114317 3639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:01.114599 3639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:01.115839 3639 hybrid_clock.cc:648] HybridClock initialized: now 1790635981115822 us; error 32 us; skew 500 ppm
Sep 28 22:53:01 dist-test-slave-56vw krb5kdc[3532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635981, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:01.118810 3639 init.cc:377] Logged in from keytab as kudu/127.16.121.129@KRBTEST.COM (short username kudu)
I20260928 22:53:01.119374 3639 webserver.cc:460] Webserver started at http://127.16.121.129:33699/ using document root <none> and password file <none>
I20260928 22:53:01.119668 3639 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:01.119735 3639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:01.119875 3639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:01.120999 3639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-0/data/instance:
uuid: "11738e7c2558412fb2f2f81c0c4e98ad"
format_stamp: "Formatted at 2026-09-28 22:53:01 on dist-test-slave-56vw"
server_key: "e0b9f3da939332c1de6a2ec4e1903ad6"
server_key_iv: "a11aceccc828b9968ae90c2a35ccc81c"
server_key_version: "encryptionkey@0"
I20260928 22:53:01.121398 3639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance:
uuid: "11738e7c2558412fb2f2f81c0c4e98ad"
format_stamp: "Formatted at 2026-09-28 22:53:01 on dist-test-slave-56vw"
server_key: "e0b9f3da939332c1de6a2ec4e1903ad6"
server_key_iv: "a11aceccc828b9968ae90c2a35ccc81c"
server_key_version: "encryptionkey@0"
I20260928 22:53:01.122843 3639 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:53:01.123978 3654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:01.124338 3639 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:53:01.124467 3639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-0/wal
uuid: "11738e7c2558412fb2f2f81c0c4e98ad"
format_stamp: "Formatted at 2026-09-28 22:53:01 on dist-test-slave-56vw"
server_key: "e0b9f3da939332c1de6a2ec4e1903ad6"
server_key_iv: "a11aceccc828b9968ae90c2a35ccc81c"
server_key_version: "encryptionkey@0"
I20260928 22:53:01.124567 3639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:01.137573 3639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:01.140165 3639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:01.140343 3639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:01.140625 3639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:53:01.140959 3639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:53:01.141014 3639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:01.141060 3639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:53:01.141084 3639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:01.147701 3639 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.129:35519
I20260928 22:53:01.147769 3767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.129:35519 every 8 connection(s)
I20260928 22:53:01.148159 3639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
Sep 28 22:53:01 dist-test-slave-56vw krb5kdc[3532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635981, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.129@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:53:01.155993 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 3639
I20260928 22:53:01.156118 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance
I20260928 22:53:01.156322 16870 external_mini_cluster.cc:1428] Setting key ca93d9f0b9b918ebf44004eecbba10fc
I20260928 22:53:01.158922 3616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:01.149044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:40561 (local address 127.16.121.190:40667)
0928 22:53:01.149122 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:01.149124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:01.149403 (+ 279us) server_negotiation.cc:407] Connection header received
0928 22:53:01.149975 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:01.149977 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:01.149988 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:01.150048 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:01.151115 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:01.151687 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:01.152835 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:01.153056 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:01.153120 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:01.155599 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:01.155601 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:01.155601 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:01.155612 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:01.157421 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:01.157853 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:01.157854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:01.157854 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:01.157914 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:01.158205 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:01.158205 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:01.158206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:01.158332 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:01.158409 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:01.158722 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 22:53:01.158828 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 22:53:01.159125 3770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:01.149002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:40667 (local address 127.16.121.129:40561)
0928 22:53:01.149211 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:01.149257 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:01.149852 (+ 595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:01.150126 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:01.150129 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:01.150726 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:01.150935 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:01.150949 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:01.151821 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:01.151822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:01.152734 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:01.152737 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:01.153155 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:01.153156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:01.153281 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:01.153902 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:01.153911 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:01.155483 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:01.157517 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:01.157526 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:01.157528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:01.157769 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:01.158001 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:01.158001 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:01.158002 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:01.158127 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:01.158399 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:01.158406 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:01.158554 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 22:53:01.158673 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 22:53:01.158850 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":116,"threads_started":1}
I20260928 22:53:01.160005 3768 heartbeater.cc:347] Connected to a master server at 127.16.121.190:40667
I20260928 22:53:01.160143 3768 heartbeater.cc:464] Registering TS with master...
I20260928 22:53:01.160375 3768 heartbeater.cc:511] Master 127.16.121.190:40667 requested a full tablet report, sending...
I20260928 22:53:01.161180 3577 ts_manager.cc:194] Registered new tserver with Master: 11738e7c2558412fb2f2f81c0c4e98ad (127.16.121.129:35519)
I20260928 22:53:01.161901 3577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.129@KRBTEST.COM'} at 127.16.121.129:40561
WARNING: no policy specified for kudu/127.16.121.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab.
Entry for principal kudu/127.16.121.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab.
I20260928 22:53:01.185560 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.130:0
--local_ip_for_outbound_sockets=127.16.121.130
--webserver_interface=127.16.121.130
--webserver_port=0
--tserver_master_addrs=127.16.121.190:40667
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab
--principal=kudu/127.16.121.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:39277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:01.268299 3773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:01.268563 3773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:01.268604 3773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:01.269843 3773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:01.269930 3773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:01.269981 3773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.130
I20260928 22:53:01.271566 3773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:39277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.130.keytab
--principal=kudu/127.16.121.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.121.130
--webserver_port=0
--tserver_master_addrs=127.16.121.190:40667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.130
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:01.271848 3773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:01.272095 3773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:01.277333 3778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:01.277343 3781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:01.277384 3779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:01.277483 3773 server_base.cc:1034] running on GCE node
I20260928 22:53:01.277921 3773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:01.278198 3773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:01.279361 3773 hybrid_clock.cc:648] HybridClock initialized: now 1790635981279336 us; error 61 us; skew 500 ppm
Sep 28 22:53:01 dist-test-slave-56vw krb5kdc[3532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635981, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:01.282105 3773 init.cc:377] Logged in from keytab as kudu/127.16.121.130@KRBTEST.COM (short username kudu)
I20260928 22:53:01.282538 3773 webserver.cc:460] Webserver started at http://127.16.121.130:45401/ using document root <none> and password file <none>
I20260928 22:53:01.282785 3773 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:01.282833 3773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:01.282982 3773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:01.284117 3773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-1/data/instance:
uuid: "b1f122d62454405d91b730c912de9b22"
format_stamp: "Formatted at 2026-09-28 22:53:01 on dist-test-slave-56vw"
server_key: "2b12b39a3aa660844aa943405fd86451"
server_key_iv: "b80b44a60511de1d449977ca7aab25ba"
server_key_version: "encryptionkey@0"
I20260928 22:53:01.284499 3773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance:
uuid: "b1f122d62454405d91b730c912de9b22"
format_stamp: "Formatted at 2026-09-28 22:53:01 on dist-test-slave-56vw"
server_key: "2b12b39a3aa660844aa943405fd86451"
server_key_iv: "b80b44a60511de1d449977ca7aab25ba"
server_key_version: "encryptionkey@0"
I20260928 22:53:01.285925 3773 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:53:01.286936 3788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:01.287249 3773 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:53:01.287372 3773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-1/wal
uuid: "b1f122d62454405d91b730c912de9b22"
format_stamp: "Formatted at 2026-09-28 22:53:01 on dist-test-slave-56vw"
server_key: "2b12b39a3aa660844aa943405fd86451"
server_key_iv: "b80b44a60511de1d449977ca7aab25ba"
server_key_version: "encryptionkey@0"
I20260928 22:53:01.287461 3773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:01.309654 3773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:01.312453 3773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:01.312634 3773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:01.312932 3773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:53:01.313267 3773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:53:01.313302 3773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:01.313326 3773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:53:01.313375 3773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:01.319630 3773 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.130:35641
I20260928 22:53:01.319695 3901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.130:35641 every 8 connection(s)
I20260928 22:53:01.320043 3773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
I20260928 22:53:01.321234 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 3773
I20260928 22:53:01.321362 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance
I20260928 22:53:01.321558 16870 external_mini_cluster.cc:1428] Setting key 013899b0108c4aae6083696a75f24e7b
Sep 28 22:53:01 dist-test-slave-56vw krb5kdc[3532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635981, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.130@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.121.131@KRBTEST.COM; defaulting to no policy
I20260928 22:53:01.332250 3616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:01.320838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:43133 (local address 127.16.121.190:40667)
0928 22:53:01.320908 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:01.320910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:01.321145 (+ 235us) server_negotiation.cc:407] Connection header received
0928 22:53:01.321864 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:01.321866 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:01.321878 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:01.321944 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:01.322968 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:01.323631 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:01.324980 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:01.325355 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:01.325444 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:01.328825 (+ 3381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:01.328828 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:01.328828 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:01.328839 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:01.330466 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:01.331182 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:01.331183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:01.331183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:01.331240 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:01.331570 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:01.331571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:01.331572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:01.331759 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:01.331820 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:01.332101 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 22:53:01.332178 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 22:53:01.332512 3904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:01.320820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:40667 (local address 127.16.121.130:43133)
0928 22:53:01.321013 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:01.321053 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:01.321756 (+ 703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:01.322026 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:01.322030 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:01.322644 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:01.322845 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:01.322849 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:01.323818 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:01.323819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:01.324850 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:01.324853 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:01.325460 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:01.325461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:01.325611 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:01.326346 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:01.326360 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:01.328652 (+ 2292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:01.330626 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:01.330641 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:01.330645 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:01.330999 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:01.331306 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:01.331307 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:01.331307 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:01.331433 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:01.331827 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:01.331832 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:01.331977 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 22:53:01.332072 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 22:53:01.332244 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":98,"threads_started":1}
I20260928 22:53:01.333141 3902 heartbeater.cc:347] Connected to a master server at 127.16.121.190:40667
I20260928 22:53:01.333266 3902 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.131@KRBTEST.COM" created.
I20260928 22:53:01.333477 3902 heartbeater.cc:511] Master 127.16.121.190:40667 requested a full tablet report, sending...
I20260928 22:53:01.334064 3577 ts_manager.cc:194] Registered new tserver with Master: b1f122d62454405d91b730c912de9b22 (127.16.121.130:35641)
I20260928 22:53:01.334532 3577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.130@KRBTEST.COM'} at 127.16.121.130:43133
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab.
Entry for principal kudu/127.16.121.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab.
I20260928 22:53:01.350620 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.131:0
--local_ip_for_outbound_sockets=127.16.121.131
--webserver_interface=127.16.121.131
--webserver_port=0
--tserver_master_addrs=127.16.121.190:40667
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab
--principal=kudu/127.16.121.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:39277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:01.433032 3907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:01.433241 3907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:01.433265 3907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:01.434432 3907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:01.434476 3907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:01.434504 3907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.131
I20260928 22:53:01.436070 3907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:39277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.131.keytab
--principal=kudu/127.16.121.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.121.131
--webserver_port=0
--tserver_master_addrs=127.16.121.190:40667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.131
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:01.436332 3907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:01.436574 3907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:01.442230 3912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:01.442245 3915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:01.442232 3913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:01.442793 3907 server_base.cc:1034] running on GCE node
I20260928 22:53:01.443017 3907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:01.443307 3907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:01.444594 3907 hybrid_clock.cc:648] HybridClock initialized: now 1790635981444577 us; error 51 us; skew 500 ppm
Sep 28 22:53:01 dist-test-slave-56vw krb5kdc[3532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635981, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:01.447572 3907 init.cc:377] Logged in from keytab as kudu/127.16.121.131@KRBTEST.COM (short username kudu)
I20260928 22:53:01.448103 3907 webserver.cc:460] Webserver started at http://127.16.121.131:33265/ using document root <none> and password file <none>
I20260928 22:53:01.448378 3907 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:01.448429 3907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:01.448566 3907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:01.449673 3907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-2/data/instance:
uuid: "408a17f86c70487fa046ba947d1f8116"
format_stamp: "Formatted at 2026-09-28 22:53:01 on dist-test-slave-56vw"
server_key: "056580daab93ecd2ffb1eb159be233dd"
server_key_iv: "36aaa00692f8508f7542447942a60808"
server_key_version: "encryptionkey@0"
I20260928 22:53:01.450054 3907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance:
uuid: "408a17f86c70487fa046ba947d1f8116"
format_stamp: "Formatted at 2026-09-28 22:53:01 on dist-test-slave-56vw"
server_key: "056580daab93ecd2ffb1eb159be233dd"
server_key_iv: "36aaa00692f8508f7542447942a60808"
server_key_version: "encryptionkey@0"
I20260928 22:53:01.451515 3907 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:53:01.452390 3922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:01.452656 3907 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:53:01.452780 3907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-2/wal
uuid: "408a17f86c70487fa046ba947d1f8116"
format_stamp: "Formatted at 2026-09-28 22:53:01 on dist-test-slave-56vw"
server_key: "056580daab93ecd2ffb1eb159be233dd"
server_key_iv: "36aaa00692f8508f7542447942a60808"
server_key_version: "encryptionkey@0"
I20260928 22:53:01.452865 3907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:01.468695 3907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:01.471355 3907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:01.471524 3907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:01.471814 3907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:53:01.472182 3907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:53:01.472244 3907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:01.472293 3907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:53:01.472326 3907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:01.479023 3907 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.131:41069
I20260928 22:53:01.479084 4035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.131:41069 every 8 connection(s)
I20260928 22:53:01.479472 3907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
Sep 28 22:53:01 dist-test-slave-56vw krb5kdc[3532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635981, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.131@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:53:01.486338 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 3907
I20260928 22:53:01.486467 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance
I20260928 22:53:01.486682 16870 external_mini_cluster.cc:1428] Setting key 2f4faaf081b9c6f8d59bc13fb1c819f7
I20260928 22:53:01.490602 3616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:01.480360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:44165 (local address 127.16.121.190:40667)
0928 22:53:01.480560 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:01.480562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:01.480747 (+ 185us) server_negotiation.cc:407] Connection header received
0928 22:53:01.481315 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:01.481317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:01.481330 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:01.481392 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:01.482445 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:01.483037 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:01.484116 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:01.484345 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:01.484426 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:01.487135 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:01.487137 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:01.487137 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:01.487146 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:01.488855 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:01.489345 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:01.489346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:01.489346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:01.489413 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:01.489827 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:01.489828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:01.489829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:01.489970 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:01.490038 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:01.490356 (+ 318us) server_negotiation.cc:299] Negotiation successful
0928 22:53:01.490480 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
I20260928 22:53:01.490712 4038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:01.480271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:40667 (local address 127.16.121.131:44165)
0928 22:53:01.480594 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:01.480655 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:01.481202 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:01.481487 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:01.481491 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:01.482117 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:01.482329 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:01.482334 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:01.483156 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:01.483157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:01.483997 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:01.484000 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:01.484443 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:01.484444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:01.484589 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:01.485207 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:01.485215 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:01.486974 (+ 1759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:01.488945 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:01.488952 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:01.488953 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:01.489224 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:01.489599 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:01.489600 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:01.489600 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:01.489728 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:01.490079 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:01.490085 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:01.490223 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 22:53:01.490320 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 22:53:01.490460 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":92,"threads_started":1}
I20260928 22:53:01.491372 4036 heartbeater.cc:347] Connected to a master server at 127.16.121.190:40667
I20260928 22:53:01.491518 4036 heartbeater.cc:464] Registering TS with master...
I20260928 22:53:01.491844 4036 heartbeater.cc:511] Master 127.16.121.190:40667 requested a full tablet report, sending...
I20260928 22:53:01.492381 3577 ts_manager.cc:194] Registered new tserver with Master: 408a17f86c70487fa046ba947d1f8116 (127.16.121.131:41069)
I20260928 22:53:01.492823 3577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.121.131@KRBTEST.COM'} at 127.16.121.131:44165
I20260928 22:53:01.498624 16870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:53:01.507366 3616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:01.500359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54174 (local address 127.16.121.190:40667)
0928 22:53:01.500471 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:01.500473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:01.500567 (+ 94us) server_negotiation.cc:407] Connection header received
0928 22:53:01.500589 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:01.500590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:01.500600 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:01.500640 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:01.501248 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:01.501813 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:01.502626 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:01.502828 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:01.502888 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:01.504239 (+ 1351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:01.504240 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:01.504240 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:01.504247 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:01.505921 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:01.506298 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:01.506299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:01.506299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:01.506359 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:01.506770 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:01.506771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:01.506771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:01.506928 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:01.507008 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:01.507177 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 22:53:01.507252 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 22:53:01.509439 3577 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54174:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:53:01.511188 3577 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:53:01.518604 4050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:01.514118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:35519 (local address 127.0.0.1:53498)
0928 22:53:01.514524 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:01.514534 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:01.514704 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:01.514973 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:01.514974 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:01.514975 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:53:01.515027 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:01.515029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:01.516295 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:01.516296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:01.517300 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:01.517303 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:01.518242 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:01.518243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:01.518338 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:01.518354 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 22:53:01.518432 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 22:53:01.518492 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":1,"thread_start_us":74,"threads_started":1}
I20260928 22:53:01.518704 4047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:01.513991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.131:41069 (local address 127.0.0.1:54454)
0928 22:53:01.514210 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:01.514247 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:01.514303 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:01.514572 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:01.514575 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:01.514577 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:53:01.514656 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:01.514658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:01.516126 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:01.516127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:01.517524 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:01.517526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:01.518394 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:01.518395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:01.518534 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:01.518538 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:53:01.518593 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 22:53:01.518636 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":74,"thread_start_us":114,"threads_started":1}
I20260928 22:53:01.519040 4051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:01.514245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54454 (local address 127.16.121.131:41069)
0928 22:53:01.514428 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:01.514433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:01.514451 (+ 18us) server_negotiation.cc:407] Connection header received
0928 22:53:01.514485 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:01.514487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:01.514499 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:01.514558 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:53:01.514745 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:01.515979 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:01.517617 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:01.518326 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:01.518393 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:01.518466 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:01.518877 (+ 411us) server_negotiation.cc:299] Negotiation successful
0928 22:53:01.518942 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":78,"threads_started":1}
I20260928 22:53:01.518652 4052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:01.514347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53498 (local address 127.16.121.129:35519)
0928 22:53:01.514509 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:01.514514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:01.514619 (+ 105us) server_negotiation.cc:407] Connection header received
0928 22:53:01.514808 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:01.514811 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:01.514828 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:01.514895 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:53:01.515150 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:01.516199 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:01.517395 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:01.518148 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:01.518221 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:01.518303 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:01.518469 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 22:53:01.518517 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":90,"threads_started":1}
I20260928 22:53:01.519728 4048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:01.513893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.130:35641 (local address 127.0.0.1:41090)
0928 22:53:01.514179 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:01.514228 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:01.514303 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:01.514997 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:01.514998 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:01.515000 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:53:01.515040 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:01.515042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:01.516028 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:01.516029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:01.517523 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:01.517526 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:01.519489 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:01.519490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:01.519553 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:01.519557 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:53:01.519617 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 22:53:01.519661 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":54,"thread_start_us":74,"threads_started":1}
I20260928 22:53:01.519819 4049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:01.514040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41090 (local address 127.16.121.130:35641)
0928 22:53:01.514754 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:01.514759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:01.514768 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:53:01.514804 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:01.514806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:01.514822 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:01.514893 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:53:01.515155 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:01.515897 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:01.518508 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:01.519312 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:01.519388 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:01.519452 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:01.519684 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 22:53:01.519725 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":78,"threads_started":1}
I20260928 22:53:01.519951 3970 tablet_service.cc:1467] Processing CreateTablet for tablet 29b64cf61bf147d98999380a93ac03e9 (DEFAULT_TABLE table=test-table [id=9e1c5367dc3c4710801e6596d749fac5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:53:01.520311 3970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29b64cf61bf147d98999380a93ac03e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:01.520598 3836 tablet_service.cc:1467] Processing CreateTablet for tablet 29b64cf61bf147d98999380a93ac03e9 (DEFAULT_TABLE table=test-table [id=9e1c5367dc3c4710801e6596d749fac5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:53:01.520897 3836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29b64cf61bf147d98999380a93ac03e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:01.522984 4053 tablet_bootstrap.cc:492] T 29b64cf61bf147d98999380a93ac03e9 P 408a17f86c70487fa046ba947d1f8116: Bootstrap starting.
I20260928 22:53:01.523681 4053 tablet_bootstrap.cc:654] T 29b64cf61bf147d98999380a93ac03e9 P 408a17f86c70487fa046ba947d1f8116: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:01.523929 4054 tablet_bootstrap.cc:492] T 29b64cf61bf147d98999380a93ac03e9 P b1f122d62454405d91b730c912de9b22: Bootstrap starting.
I20260928 22:53:01.523973 3702 tablet_service.cc:1467] Processing CreateTablet for tablet 29b64cf61bf147d98999380a93ac03e9 (DEFAULT_TABLE table=test-table [id=9e1c5367dc3c4710801e6596d749fac5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:53:01.524338 3702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29b64cf61bf147d98999380a93ac03e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:01.524403 4053 log.cc:824] T 29b64cf61bf147d98999380a93ac03e9 P 408a17f86c70487fa046ba947d1f8116: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:01.524724 4054 tablet_bootstrap.cc:654] T 29b64cf61bf147d98999380a93ac03e9 P b1f122d62454405d91b730c912de9b22: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:01.525230 4053 tablet_bootstrap.cc:492] T 29b64cf61bf147d98999380a93ac03e9 P 408a17f86c70487fa046ba947d1f8116: No bootstrap required, opened a new log
I20260928 22:53:01.525309 4053 ts_tablet_manager.cc:1397] T 29b64cf61bf147d98999380a93ac03e9 P 408a17f86c70487fa046ba947d1f8116: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 22:53:01.525300 4054 log.cc:824] T 29b64cf61bf147d98999380a93ac03e9 P b1f122d62454405d91b730c912de9b22: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:01.526084 4054 tablet_bootstrap.cc:492] T 29b64cf61bf147d98999380a93ac03e9 P b1f122d62454405d91b730c912de9b22: No bootstrap required, opened a new log
I20260928 22:53:01.526154 4054 ts_tablet_manager.cc:1397] T 29b64cf61bf147d98999380a93ac03e9 P b1f122d62454405d91b730c912de9b22: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:53:01.527302 4057 tablet_bootstrap.cc:492] T 29b64cf61bf147d98999380a93ac03e9 P 11738e7c2558412fb2f2f81c0c4e98ad: Bootstrap starting.
I20260928 22:53:01.527463 4053 raft_consensus.cc:348] T 29b64cf61bf147d98999380a93ac03e9 P 408a17f86c70487fa046ba947d1f8116 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f122d62454405d91b730c912de9b22" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 35641 } } peers { permanent_uuid: "408a17f86c70487fa046ba947d1f8116" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41069 } } peers { permanent_uuid: "11738e7c2558412fb2f2f81c0c4e98ad" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 35519 } }
I20260928 22:53:01.527729 4053 raft_consensus.cc:374] T 29b64cf61bf147d98999380a93ac03e9 P 408a17f86c70487fa046ba947d1f8116 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:01.527773 4053 raft_consensus.cc:729] T 29b64cf61bf147d98999380a93ac03e9 P 408a17f86c70487fa046ba947d1f8116 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 408a17f86c70487fa046ba947d1f8116, State: Initialized, Role: FOLLOWER
I20260928 22:53:01.527901 4053 consensus_queue.cc:260] T 29b64cf61bf147d98999380a93ac03e9 P 408a17f86c70487fa046ba947d1f8116 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f122d62454405d91b730c912de9b22" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 35641 } } peers { permanent_uuid: "408a17f86c70487fa046ba947d1f8116" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41069 } } peers { permanent_uuid: "11738e7c2558412fb2f2f81c0c4e98ad" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 35519 } }
I20260928 22:53:01.528096 4057 tablet_bootstrap.cc:654] T 29b64cf61bf147d98999380a93ac03e9 P 11738e7c2558412fb2f2f81c0c4e98ad: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:01.528192 4053 ts_tablet_manager.cc:1428] T 29b64cf61bf147d98999380a93ac03e9 P 408a17f86c70487fa046ba947d1f8116: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 22:53:01.528321 4036 heartbeater.cc:503] Master 127.16.121.190:40667 was elected leader, sending a full tablet report...
I20260928 22:53:01.528537 4054 raft_consensus.cc:348] T 29b64cf61bf147d98999380a93ac03e9 P b1f122d62454405d91b730c912de9b22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f122d62454405d91b730c912de9b22" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 35641 } } peers { permanent_uuid: "408a17f86c70487fa046ba947d1f8116" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41069 } } peers { permanent_uuid: "11738e7c2558412fb2f2f81c0c4e98ad" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 35519 } }
I20260928 22:53:01.528707 4054 raft_consensus.cc:374] T 29b64cf61bf147d98999380a93ac03e9 P b1f122d62454405d91b730c912de9b22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:01.528748 4054 raft_consensus.cc:729] T 29b64cf61bf147d98999380a93ac03e9 P b1f122d62454405d91b730c912de9b22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1f122d62454405d91b730c912de9b22, State: Initialized, Role: FOLLOWER
I20260928 22:53:01.528795 4057 log.cc:824] T 29b64cf61bf147d98999380a93ac03e9 P 11738e7c2558412fb2f2f81c0c4e98ad: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:01.528855 4054 consensus_queue.cc:260] T 29b64cf61bf147d98999380a93ac03e9 P b1f122d62454405d91b730c912de9b22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f122d62454405d91b730c912de9b22" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 35641 } } peers { permanent_uuid: "408a17f86c70487fa046ba947d1f8116" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41069 } } peers { permanent_uuid: "11738e7c2558412fb2f2f81c0c4e98ad" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 35519 } }
I20260928 22:53:01.529130 3902 heartbeater.cc:503] Master 127.16.121.190:40667 was elected leader, sending a full tablet report...
I20260928 22:53:01.529129 4054 ts_tablet_manager.cc:1428] T 29b64cf61bf147d98999380a93ac03e9 P b1f122d62454405d91b730c912de9b22: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:53:01.529699 4057 tablet_bootstrap.cc:492] T 29b64cf61bf147d98999380a93ac03e9 P 11738e7c2558412fb2f2f81c0c4e98ad: No bootstrap required, opened a new log
I20260928 22:53:01.529767 4057 ts_tablet_manager.cc:1397] T 29b64cf61bf147d98999380a93ac03e9 P 11738e7c2558412fb2f2f81c0c4e98ad: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 22:53:01.531337 4057 raft_consensus.cc:348] T 29b64cf61bf147d98999380a93ac03e9 P 11738e7c2558412fb2f2f81c0c4e98ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f122d62454405d91b730c912de9b22" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 35641 } } peers { permanent_uuid: "408a17f86c70487fa046ba947d1f8116" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41069 } } peers { permanent_uuid: "11738e7c2558412fb2f2f81c0c4e98ad" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 35519 } }
I20260928 22:53:01.531463 4057 raft_consensus.cc:374] T 29b64cf61bf147d98999380a93ac03e9 P 11738e7c2558412fb2f2f81c0c4e98ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:01.531484 4057 raft_consensus.cc:729] T 29b64cf61bf147d98999380a93ac03e9 P 11738e7c2558412fb2f2f81c0c4e98ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11738e7c2558412fb2f2f81c0c4e98ad, State: Initialized, Role: FOLLOWER
I20260928 22:53:01.531630 4057 consensus_queue.cc:260] T 29b64cf61bf147d98999380a93ac03e9 P 11738e7c2558412fb2f2f81c0c4e98ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f122d62454405d91b730c912de9b22" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 35641 } } peers { permanent_uuid: "408a17f86c70487fa046ba947d1f8116" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41069 } } peers { permanent_uuid: "11738e7c2558412fb2f2f81c0c4e98ad" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 35519 } }
I20260928 22:53:01.531924 4057 ts_tablet_manager.cc:1428] T 29b64cf61bf147d98999380a93ac03e9 P 11738e7c2558412fb2f2f81c0c4e98ad: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 22:53:01.531939 3768 heartbeater.cc:503] Master 127.16.121.190:40667 was elected leader, sending a full tablet report...
W20260928 22:53:01.570675 3903 tablet.cc:2369] T 29b64cf61bf147d98999380a93ac03e9 P b1f122d62454405d91b730c912de9b22: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 22:53:01.648978 3769 tablet.cc:2369] T 29b64cf61bf147d98999380a93ac03e9 P 11738e7c2558412fb2f2f81c0c4e98ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:53:01.694325 4061 raft_consensus.cc:482] T 29b64cf61bf147d98999380a93ac03e9 P 11738e7c2558412fb2f2f81c0c4e98ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:53:01.694504 4061 raft_consensus.cc:504] T 29b64cf61bf147d98999380a93ac03e9 P 11738e7c2558412fb2f2f81c0c4e98ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f122d62454405d91b730c912de9b22" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 35641 } } peers { permanent_uuid: "408a17f86c70487fa046ba947d1f8116" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41069 } } peers { permanent_uuid: "11738e7c2558412fb2f2f81c0c4e98ad" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 35519 } }
I20260928 22:53:01.695031 4061 leader_election.cc:288] T 29b64cf61bf147d98999380a93ac03e9 P 11738e7c2558412fb2f2f81c0c4e98ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b1f122d62454405d91b730c912de9b22 (127.16.121.130:35641), 408a17f86c70487fa046ba947d1f8116 (127.16.121.131:41069)
I20260928 22:53:01.699854 4049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:01.695329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:43721 (local address 127.16.121.130:35641)
0928 22:53:01.695382 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:01.695384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:01.695589 (+ 205us) server_negotiation.cc:407] Connection header received
0928 22:53:01.695623 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:01.695624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:01.695638 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:01.695705 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:53:01.695939 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:01.696750 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:01.698567 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:01.699327 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:01.699398 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:01.699408 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:01.699705 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 22:53:01.699778 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 22:53:01.699878 4062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:01.695294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.130:35641 (local address 127.16.121.129:43721)
0928 22:53:01.695444 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:01.695458 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:01.695539 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:01.695725 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:01.695726 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:01.695729 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:53:01.695848 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:01.695851 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:01.696843 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:01.696844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:01.698431 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:01.698436 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:01.699464 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:01.699466 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:01.699563 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:01.699570 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 22:53:01.699647 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 22:53:01.699733 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":71,"thread_start_us":106,"threads_started":1}
I20260928 22:53:01.699965 4051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:01.695384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:51321 (local address 127.16.121.131:41069)
0928 22:53:01.695436 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:01.695438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:01.695586 (+ 148us) server_negotiation.cc:407] Connection header received
0928 22:53:01.695623 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:01.695624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:01.695635 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:01.695705 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:53:01.695945 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:01.696700 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:01.698568 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:01.699298 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:01.699397 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:01.699408 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:01.699824 (+ 416us) server_negotiation.cc:299] Negotiation successful
0928 22:53:01.699880 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 22:53:01.700338 4063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:01.695289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.131:41069 (local address 127.16.121.129:51321)
0928 22:53:01.695453 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:01.695463 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:01.695539 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:01.695725 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:01.695726 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:01.695729 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:53:01.695853 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:01.695855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:01.696842 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:01.696844 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:01.698431 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:01.698436 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:01.699468 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:01.699469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:01.699599 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:01.699606 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 22:53:01.699786 (+ 180us) client_negotiation.cc:240] Negotiation successful
0928 22:53:01.699836 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":85,"thread_start_us":77,"threads_started":1}
I20260928 22:53:01.700372 3856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b64cf61bf147d98999380a93ac03e9" candidate_uuid: "11738e7c2558412fb2f2f81c0c4e98ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1f122d62454405d91b730c912de9b22" is_pre_election: true
I20260928 22:53:01.700620 3856 raft_consensus.cc:2454] T 29b64cf61bf147d98999380a93ac03e9 P b1f122d62454405d91b730c912de9b22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11738e7c2558412fb2f2f81c0c4e98ad in term 0.
I20260928 22:53:01.700855 3990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b64cf61bf147d98999380a93ac03e9" candidate_uuid: "11738e7c2558412fb2f2f81c0c4e98ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "408a17f86c70487fa046ba947d1f8116" is_pre_election: true
I20260928 22:53:01.700902 3656 leader_election.cc:302] T 29b64cf61bf147d98999380a93ac03e9 P 11738e7c2558412fb2f2f81c0c4e98ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11738e7c2558412fb2f2f81c0c4e98ad, b1f122d62454405d91b730c912de9b22; no voters:
I20260928 22:53:01.700973 3990 raft_consensus.cc:2454] T 29b64cf61bf147d98999380a93ac03e9 P 408a17f86c70487fa046ba947d1f8116 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11738e7c2558412fb2f2f81c0c4e98ad in term 0.
I20260928 22:53:01.701081 4061 raft_consensus.cc:2781] T 29b64cf61bf147d98999380a93ac03e9 P 11738e7c2558412fb2f2f81c0c4e98ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:53:01.701144 4061 raft_consensus.cc:482] T 29b64cf61bf147d98999380a93ac03e9 P 11738e7c2558412fb2f2f81c0c4e98ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:53:01.701175 4061 raft_consensus.cc:3037] T 29b64cf61bf147d98999380a93ac03e9 P 11738e7c2558412fb2f2f81c0c4e98ad [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:01.701942 4061 raft_consensus.cc:504] T 29b64cf61bf147d98999380a93ac03e9 P 11738e7c2558412fb2f2f81c0c4e98ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f122d62454405d91b730c912de9b22" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 35641 } } peers { permanent_uuid: "408a17f86c70487fa046ba947d1f8116" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41069 } } peers { permanent_uuid: "11738e7c2558412fb2f2f81c0c4e98ad" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 35519 } }
I20260928 22:53:01.702107 4061 leader_election.cc:288] T 29b64cf61bf147d98999380a93ac03e9 P 11738e7c2558412fb2f2f81c0c4e98ad [CANDIDATE]: Term 1 election: Requested vote from peers b1f122d62454405d91b730c912de9b22 (127.16.121.130:35641), 408a17f86c70487fa046ba947d1f8116 (127.16.121.131:41069)
I20260928 22:53:01.702340 3856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b64cf61bf147d98999380a93ac03e9" candidate_uuid: "11738e7c2558412fb2f2f81c0c4e98ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1f122d62454405d91b730c912de9b22"
I20260928 22:53:01.702419 3856 raft_consensus.cc:3037] T 29b64cf61bf147d98999380a93ac03e9 P b1f122d62454405d91b730c912de9b22 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:01.702384 3990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b64cf61bf147d98999380a93ac03e9" candidate_uuid: "11738e7c2558412fb2f2f81c0c4e98ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "408a17f86c70487fa046ba947d1f8116"
I20260928 22:53:01.702483 3990 raft_consensus.cc:3037] T 29b64cf61bf147d98999380a93ac03e9 P 408a17f86c70487fa046ba947d1f8116 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:01.703307 3990 raft_consensus.cc:2454] T 29b64cf61bf147d98999380a93ac03e9 P 408a17f86c70487fa046ba947d1f8116 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11738e7c2558412fb2f2f81c0c4e98ad in term 1.
I20260928 22:53:01.703410 3856 raft_consensus.cc:2454] T 29b64cf61bf147d98999380a93ac03e9 P b1f122d62454405d91b730c912de9b22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11738e7c2558412fb2f2f81c0c4e98ad in term 1.
I20260928 22:53:01.703498 3655 leader_election.cc:302] T 29b64cf61bf147d98999380a93ac03e9 P 11738e7c2558412fb2f2f81c0c4e98ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11738e7c2558412fb2f2f81c0c4e98ad, 408a17f86c70487fa046ba947d1f8116; no voters:
I20260928 22:53:01.703617 4061 raft_consensus.cc:2781] T 29b64cf61bf147d98999380a93ac03e9 P 11738e7c2558412fb2f2f81c0c4e98ad [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:53:01.703783 4061 raft_consensus.cc:686] T 29b64cf61bf147d98999380a93ac03e9 P 11738e7c2558412fb2f2f81c0c4e98ad [term 1 LEADER]: Becoming Leader. State: Replica: 11738e7c2558412fb2f2f81c0c4e98ad, State: Running, Role: LEADER
I20260928 22:53:01.703893 4061 consensus_queue.cc:237] T 29b64cf61bf147d98999380a93ac03e9 P 11738e7c2558412fb2f2f81c0c4e98ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f122d62454405d91b730c912de9b22" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 35641 } } peers { permanent_uuid: "408a17f86c70487fa046ba947d1f8116" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41069 } } peers { permanent_uuid: "11738e7c2558412fb2f2f81c0c4e98ad" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 35519 } }
I20260928 22:53:01.704862 3577 catalog_manager.cc:5495] T 29b64cf61bf147d98999380a93ac03e9 P 11738e7c2558412fb2f2f81c0c4e98ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 11738e7c2558412fb2f2f81c0c4e98ad (127.16.121.129). New cstate: current_term: 1 leader_uuid: "11738e7c2558412fb2f2f81c0c4e98ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f122d62454405d91b730c912de9b22" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 35641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "408a17f86c70487fa046ba947d1f8116" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11738e7c2558412fb2f2f81c0c4e98ad" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 35519 } health_report { overall_health: HEALTHY } } }
W20260928 22:53:01.730161 4037 tablet.cc:2369] T 29b64cf61bf147d98999380a93ac03e9 P 408a17f86c70487fa046ba947d1f8116: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 22:53:01.748317 4052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:01.745586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53502 (local address 127.16.121.129:35519)
0928 22:53:01.745656 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:01.745658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:01.745667 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:53:01.745705 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:01.745706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:01.745728 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:01.745791 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:53:01.745939 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:01.746420 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:01.747491 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:01.747692 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:01.747760 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:01.747934 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:53:01.748059 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:53:01.748125 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:01.748177 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 22:53:01.748197 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 22:53:01.750422 3990 raft_consensus.cc:1261] T 29b64cf61bf147d98999380a93ac03e9 P 408a17f86c70487fa046ba947d1f8116 [term 1 FOLLOWER]: Refusing update from remote peer 11738e7c2558412fb2f2f81c0c4e98ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:53:01.750422 3856 raft_consensus.cc:1261] T 29b64cf61bf147d98999380a93ac03e9 P b1f122d62454405d91b730c912de9b22 [term 1 FOLLOWER]: Refusing update from remote peer 11738e7c2558412fb2f2f81c0c4e98ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:53:01.750828 4061 consensus_queue.cc:1035] T 29b64cf61bf147d98999380a93ac03e9 P 11738e7c2558412fb2f2f81c0c4e98ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1f122d62454405d91b730c912de9b22" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 35641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:53:01.751005 4064 consensus_queue.cc:1035] T 29b64cf61bf147d98999380a93ac03e9 P 11738e7c2558412fb2f2f81c0c4e98ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "408a17f86c70487fa046ba947d1f8116" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 41069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:53:01.754127 4067 mvcc.cc:205] Tried to move back new op lower bound from 7334444981246513152 to 7334444981059854336. Current Snapshot: MvccSnapshot[applied={T|T < 7334444981246513152 or (T in {7334444981246513152})}]
I20260928 22:53:01.754392 4068 mvcc.cc:205] Tried to move back new op lower bound from 7334444981246513152 to 7334444981059854336. Current Snapshot: MvccSnapshot[applied={T|T < 7334444981246513152}]
I20260928 22:53:01.755944 3576 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:53:01.756049 3576 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:53:01.756846 3576 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1fc3de5a78d24b699d99fb31468fbae5: Sending DeleteTablet for 3 replicas of tablet 29b64cf61bf147d98999380a93ac03e9
I20260928 22:53:01.757181 3970 tablet_service.cc:1514] Processing DeleteTablet for tablet 29b64cf61bf147d98999380a93ac03e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:53:01 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:54454
I20260928 22:53:01.757208 3836 tablet_service.cc:1514] Processing DeleteTablet for tablet 29b64cf61bf147d98999380a93ac03e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:53:01 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:41090
I20260928 22:53:01.757486 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 3639
I20260928 22:53:01.757534 3702 tablet_service.cc:1514] Processing DeleteTablet for tablet 29b64cf61bf147d98999380a93ac03e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:53:01 UTC) from {username='kudu', principal='kudu/127.16.121.190@KRBTEST.COM'} at 127.0.0.1:53498
I20260928 22:53:01.758121 4076 tablet_replica.cc:330] T 29b64cf61bf147d98999380a93ac03e9 P b1f122d62454405d91b730c912de9b22: stopping tablet replica
I20260928 22:53:01.758249 4076 raft_consensus.cc:2229] T 29b64cf61bf147d98999380a93ac03e9 P b1f122d62454405d91b730c912de9b22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:53:01.758379 4076 raft_consensus.cc:2258] T 29b64cf61bf147d98999380a93ac03e9 P b1f122d62454405d91b730c912de9b22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:53:01.758585 4077 tablet_replica.cc:330] T 29b64cf61bf147d98999380a93ac03e9 P 408a17f86c70487fa046ba947d1f8116: stopping tablet replica
I20260928 22:53:01.758692 4077 raft_consensus.cc:2229] T 29b64cf61bf147d98999380a93ac03e9 P 408a17f86c70487fa046ba947d1f8116 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:53:01.758802 4077 raft_consensus.cc:2258] T 29b64cf61bf147d98999380a93ac03e9 P 408a17f86c70487fa046ba947d1f8116 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:53:01.758925 4076 ts_tablet_manager.cc:1905] T 29b64cf61bf147d98999380a93ac03e9 P b1f122d62454405d91b730c912de9b22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:53:01.760756 4076 ts_tablet_manager.cc:1918] T 29b64cf61bf147d98999380a93ac03e9 P b1f122d62454405d91b730c912de9b22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:53:01.760834 4076 log.cc:1196] T 29b64cf61bf147d98999380a93ac03e9 P b1f122d62454405d91b730c912de9b22: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-1/wal/wals/29b64cf61bf147d98999380a93ac03e9
I20260928 22:53:01.760893 4077 ts_tablet_manager.cc:1905] T 29b64cf61bf147d98999380a93ac03e9 P 408a17f86c70487fa046ba947d1f8116: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:53:01.761142 4076 ts_tablet_manager.cc:1939] T 29b64cf61bf147d98999380a93ac03e9 P b1f122d62454405d91b730c912de9b22: Deleting consensus metadata
I20260928 22:53:01.761552 3562 catalog_manager.cc:4841] TS b1f122d62454405d91b730c912de9b22 (127.16.121.130:35641): tablet 29b64cf61bf147d98999380a93ac03e9 (table test-table [id=9e1c5367dc3c4710801e6596d749fac5]) successfully deleted
I20260928 22:53:01.762719 4077 ts_tablet_manager.cc:1918] T 29b64cf61bf147d98999380a93ac03e9 P 408a17f86c70487fa046ba947d1f8116: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:53:01.762810 4077 log.cc:1196] T 29b64cf61bf147d98999380a93ac03e9 P 408a17f86c70487fa046ba947d1f8116: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790635733329745-16870-0/minicluster-data/ts-2/wal/wals/29b64cf61bf147d98999380a93ac03e9
I20260928 22:53:01.763103 4077 ts_tablet_manager.cc:1939] T 29b64cf61bf147d98999380a93ac03e9 P 408a17f86c70487fa046ba947d1f8116: Deleting consensus metadata
I20260928 22:53:01.763541 3561 catalog_manager.cc:4841] TS 408a17f86c70487fa046ba947d1f8116 (127.16.121.131:41069): tablet 29b64cf61bf147d98999380a93ac03e9 (table test-table [id=9e1c5367dc3c4710801e6596d749fac5]) successfully deleted
W20260928 22:53:01.763764 3562 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.121.129:35519 (error 108)
I20260928 22:53:01.764348 4048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:01.764195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:35519 (local address 127.0.0.1:53508)
0928 22:53:01.764258 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:01.764307 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.129:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 22:53:01.764485 3562 catalog_manager.cc:4584] TS 11738e7c2558412fb2f2f81c0c4e98ad (127.16.121.129:35519): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 29b64cf61bf147d98999380a93ac03e9: Network error: Client connection negotiation failed: client connection to 127.16.121.129:35519: connect: Connection refused (error 111)
I20260928 22:53:01.764519 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 3773
I20260928 22:53:01.773232 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 3907
I20260928 22:53:01.778231 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 3545
2026-09-28T22:53:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2159 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T22:53:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:53:01Z Disabled control of system clock
I20260928 22:53:01.807499 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:38769
--webserver_interface=127.16.121.190
--webserver_port=0
--builtin_ntp_servers=127.16.121.148:45863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:38769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 22:53:01.895892 4082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:01.896180 4082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:01.896217 4082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:01.897454 4082 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:53:01.897531 4082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:01.897553 4082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:01.897584 4082 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:53:01.897634 4082 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:53:01.899165 4082 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:45863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:38769
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:38769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.121.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:01.899483 4082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:01.899955 4082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:01.905217 4088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:01.905230 4090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:01.905421 4082 server_base.cc:1034] running on GCE node
W20260928 22:53:01.905272 4087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:01.905973 4082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:01.906349 4082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:01.907562 4082 hybrid_clock.cc:648] HybridClock initialized: now 1790635981907550 us; error 55 us; skew 500 ppm
I20260928 22:53:01.909030 4082 webserver.cc:460] Webserver started at http://127.16.121.190:45817/ using document root <none> and password file <none>
I20260928 22:53:01.909313 4082 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:01.909391 4082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:01.909610 4082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:01.910753 4082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "446aea124f984ab9be364232dbf7f202"
format_stamp: "Formatted at 2026-09-28 22:53:01 on dist-test-slave-56vw"
server_key: "1bca36f043309820da209a10ab662c2b"
server_key_iv: "35939143357821dea19ca9ba6cb4c065"
server_key_version: "encryptionkey@0"
I20260928 22:53:01.911207 4082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "446aea124f984ab9be364232dbf7f202"
format_stamp: "Formatted at 2026-09-28 22:53:01 on dist-test-slave-56vw"
server_key: "1bca36f043309820da209a10ab662c2b"
server_key_iv: "35939143357821dea19ca9ba6cb4c065"
server_key_version: "encryptionkey@0"
I20260928 22:53:01.912709 4082 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:53:01.913786 4096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:01.914068 4082 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:53:01.914166 4082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "446aea124f984ab9be364232dbf7f202"
format_stamp: "Formatted at 2026-09-28 22:53:01 on dist-test-slave-56vw"
server_key: "1bca36f043309820da209a10ab662c2b"
server_key_iv: "35939143357821dea19ca9ba6cb4c065"
server_key_version: "encryptionkey@0"
I20260928 22:53:01.914247 4082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:01.934890 4082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:01.935307 4082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:01.935428 4082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:01.939745 4082 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.190:38769
I20260928 22:53:01.939788 4148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.190:38769 every 8 connection(s)
I20260928 22:53:01.940272 4082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:53:01.941715 4149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:01.943576 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 4082
I20260928 22:53:01.943682 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:53:01.943897 16870 external_mini_cluster.cc:1428] Setting key 31e01cda691ab20af00ab03a814c0601
I20260928 22:53:01.945024 4149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 446aea124f984ab9be364232dbf7f202: Bootstrap starting.
I20260928 22:53:01.945616 4149 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 446aea124f984ab9be364232dbf7f202: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:01.945902 4149 log.cc:824] T 00000000000000000000000000000000 P 446aea124f984ab9be364232dbf7f202: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:01.946614 4149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 446aea124f984ab9be364232dbf7f202: No bootstrap required, opened a new log
I20260928 22:53:01.948443 4149 raft_consensus.cc:348] T 00000000000000000000000000000000 P 446aea124f984ab9be364232dbf7f202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "446aea124f984ab9be364232dbf7f202" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 38769 } }
I20260928 22:53:01.948603 4149 raft_consensus.cc:374] T 00000000000000000000000000000000 P 446aea124f984ab9be364232dbf7f202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:01.948626 4149 raft_consensus.cc:729] T 00000000000000000000000000000000 P 446aea124f984ab9be364232dbf7f202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 446aea124f984ab9be364232dbf7f202, State: Initialized, Role: FOLLOWER
I20260928 22:53:01.948757 4149 consensus_queue.cc:260] T 00000000000000000000000000000000 P 446aea124f984ab9be364232dbf7f202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "446aea124f984ab9be364232dbf7f202" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 38769 } }
I20260928 22:53:01.948872 4149 raft_consensus.cc:388] T 00000000000000000000000000000000 P 446aea124f984ab9be364232dbf7f202 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:53:01.948904 4149 raft_consensus.cc:482] T 00000000000000000000000000000000 P 446aea124f984ab9be364232dbf7f202 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:53:01.948985 4149 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 446aea124f984ab9be364232dbf7f202 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:01.949772 4149 raft_consensus.cc:504] T 00000000000000000000000000000000 P 446aea124f984ab9be364232dbf7f202 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "446aea124f984ab9be364232dbf7f202" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 38769 } }
I20260928 22:53:01.949887 4149 leader_election.cc:302] T 00000000000000000000000000000000 P 446aea124f984ab9be364232dbf7f202 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 446aea124f984ab9be364232dbf7f202; no voters:
I20260928 22:53:01.950376 4149 leader_election.cc:288] T 00000000000000000000000000000000 P 446aea124f984ab9be364232dbf7f202 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:53:01.950486 4154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 446aea124f984ab9be364232dbf7f202 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:53:01.950659 4149 sys_catalog.cc:564] T 00000000000000000000000000000000 P 446aea124f984ab9be364232dbf7f202 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:53:01.950690 4154 raft_consensus.cc:686] T 00000000000000000000000000000000 P 446aea124f984ab9be364232dbf7f202 [term 1 LEADER]: Becoming Leader. State: Replica: 446aea124f984ab9be364232dbf7f202, State: Running, Role: LEADER
I20260928 22:53:01.950903 4154 consensus_queue.cc:237] T 00000000000000000000000000000000 P 446aea124f984ab9be364232dbf7f202 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "446aea124f984ab9be364232dbf7f202" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 38769 } }
I20260928 22:53:01.951658 4155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 446aea124f984ab9be364232dbf7f202 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "446aea124f984ab9be364232dbf7f202" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "446aea124f984ab9be364232dbf7f202" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 38769 } } }
I20260928 22:53:01.951745 4155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 446aea124f984ab9be364232dbf7f202 [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:01.951895 4156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 446aea124f984ab9be364232dbf7f202 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 446aea124f984ab9be364232dbf7f202. Latest consensus state: current_term: 1 leader_uuid: "446aea124f984ab9be364232dbf7f202" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "446aea124f984ab9be364232dbf7f202" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 38769 } } }
I20260928 22:53:01.951944 4156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 446aea124f984ab9be364232dbf7f202 [sys.catalog]: This master's current role is: LEADER
W20260928 22:53:01.953217 4169 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 446aea124f984ab9be364232dbf7f202: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:53:01.953274 4169 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:53:01.953317 4166 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:53:01.953467 4166 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:53:01.954911 4152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:01.944762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57416 (local address 127.16.121.190:38769)
0928 22:53:01.948895 (+ 4133us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:01.948906 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:01.948959 (+ 53us) server_negotiation.cc:407] Connection header received
0928 22:53:01.949112 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:01.949121 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:01.949199 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:01.949374 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:01.949716 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:01.951295 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:01.952536 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:01.953185 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:01.953324 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:01.954127 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:01.954132 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:01.954132 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:53:01.954336 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:01.954373 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:53:01.954384 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:01.954480 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:01.954555 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 22:53:01.954643 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4045,"thread_start_us":67,"threads_started":1}
I20260928 22:53:01.955299 4166 catalog_manager.cc:1347] Generated new cluster ID: 790b08f6150142ba829788552b4fb7d7
I20260928 22:53:01.955344 4166 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:53:01.977766 4166 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:53:01.978535 4166 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:53:01.982657 4166 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 446aea124f984ab9be364232dbf7f202: Generated new TSK 0
I20260928 22:53:01.982872 4166 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:53:01.987305 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.129:0
--local_ip_for_outbound_sockets=127.16.121.129
--webserver_interface=127.16.121.129
--webserver_port=0
--tserver_master_addrs=127.16.121.190:38769
--builtin_ntp_servers=127.16.121.148:45863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 22:53:02.078143 4173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:02.078382 4173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:02.078418 4173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:02.079634 4173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:02.079691 4173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:02.079732 4173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.129
I20260928 22:53:02.081086 4173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:45863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.121.129
--webserver_port=0
--tserver_master_addrs=127.16.121.190:38769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.129
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:02.081350 4173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:02.081652 4173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 22:53:02.088258 4173 server_base.cc:1034] running on GCE node
W20260928 22:53:02.088393 4178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:02.088239 4181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:02.088223 4179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:02.088754 4173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:02.089022 4173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:02.090210 4173 hybrid_clock.cc:648] HybridClock initialized: now 1790635982090191 us; error 44 us; skew 500 ppm
I20260928 22:53:02.091668 4173 webserver.cc:460] Webserver started at http://127.16.121.129:44459/ using document root <none> and password file <none>
I20260928 22:53:02.091943 4173 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:02.092020 4173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:02.092170 4173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:02.093300 4173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-0/data/instance:
uuid: "4941053186314a97b42634746ed3dc32"
format_stamp: "Formatted at 2026-09-28 22:53:02 on dist-test-slave-56vw"
server_key: "ae5c44e713cb87dcf64cd8a407a54d2d"
server_key_iv: "0af8efc1593c4768864a7a16f584c051"
server_key_version: "encryptionkey@0"
I20260928 22:53:02.093791 4173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance:
uuid: "4941053186314a97b42634746ed3dc32"
format_stamp: "Formatted at 2026-09-28 22:53:02 on dist-test-slave-56vw"
server_key: "ae5c44e713cb87dcf64cd8a407a54d2d"
server_key_iv: "0af8efc1593c4768864a7a16f584c051"
server_key_version: "encryptionkey@0"
I20260928 22:53:02.095274 4173 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:53:02.096109 4187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:02.096302 4173 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:53:02.096421 4173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-0/wal
uuid: "4941053186314a97b42634746ed3dc32"
format_stamp: "Formatted at 2026-09-28 22:53:02 on dist-test-slave-56vw"
server_key: "ae5c44e713cb87dcf64cd8a407a54d2d"
server_key_iv: "0af8efc1593c4768864a7a16f584c051"
server_key_version: "encryptionkey@0"
I20260928 22:53:02.096503 4173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:02.135986 4173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:02.136343 4173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:02.136516 4173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:02.136786 4173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:53:02.137141 4173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:53:02.137197 4173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:02.137238 4173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:53:02.137269 4173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:02.143216 4173 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.129:42945
I20260928 22:53:02.143386 4300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.129:42945 every 8 connection(s)
I20260928 22:53:02.143697 4173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-0/data/info.pb
I20260928 22:53:02.144409 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 4173
I20260928 22:53:02.144523 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-0/wal/instance
I20260928 22:53:02.144752 16870 external_mini_cluster.cc:1428] Setting key 84766ecd39e1adf6dc66f28e2d8f6707
I20260928 22:53:02.145850 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.130:0
--local_ip_for_outbound_sockets=127.16.121.130
--webserver_interface=127.16.121.130
--webserver_port=0
--tserver_master_addrs=127.16.121.190:38769
--builtin_ntp_servers=127.16.121.148:45863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 22:53:02.149971 4152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:02.144536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:40979 (local address 127.16.121.190:38769)
0928 22:53:02.144597 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:02.144599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:02.144902 (+ 303us) server_negotiation.cc:407] Connection header received
0928 22:53:02.145595 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:02.145596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:02.145614 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:02.145661 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:02.146656 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:02.147297 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:02.148312 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:02.148531 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:02.148620 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:02.149635 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:02.149637 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:02.149637 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:53:02.149643 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:02.149659 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:53:02.149664 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:02.149736 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:02.149837 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 22:53:02.149879 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 22:53:02.150195 4303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:02.144553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:38769 (local address 127.16.121.129:40979)
0928 22:53:02.144763 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:02.144815 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:02.145453 (+ 638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:02.145744 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:02.145748 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:02.145759 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:02.146505 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:02.146513 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:02.147407 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:02.147408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:02.148213 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:02.148215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:02.148632 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:02.148633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:02.148783 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:02.149374 (+ 591us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:53:02.149375 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:02.149487 (+ 112us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:53:02.149515 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:02.149739 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:02.149746 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:02.149749 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:53:02.149841 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 22:53:02.149952 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":90,"threads_started":1}
I20260928 22:53:02.150768 4301 heartbeater.cc:347] Connected to a master server at 127.16.121.190:38769
I20260928 22:53:02.150897 4301 heartbeater.cc:464] Registering TS with master...
I20260928 22:53:02.151188 4301 heartbeater.cc:511] Master 127.16.121.190:38769 requested a full tablet report, sending...
I20260928 22:53:02.151881 4113 ts_manager.cc:194] Registered new tserver with Master: 4941053186314a97b42634746ed3dc32 (127.16.121.129:42945)
I20260928 22:53:02.152693 4113 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.121.129:40979
W20260928 22:53:02.230718 4304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:02.230938 4304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:02.230963 4304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:02.232292 4304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:02.232344 4304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:02.232371 4304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.130
I20260928 22:53:02.233806 4304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:45863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.121.130
--webserver_port=0
--tserver_master_addrs=127.16.121.190:38769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.130
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:02.234094 4304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:02.234445 4304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:02.239934 4310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:02.240002 4309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:02.239943 4312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:02.240314 4304 server_base.cc:1034] running on GCE node
I20260928 22:53:02.240572 4304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:02.240878 4304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:02.242050 4304 hybrid_clock.cc:648] HybridClock initialized: now 1790635982242022 us; error 40 us; skew 500 ppm
I20260928 22:53:02.243534 4304 webserver.cc:460] Webserver started at http://127.16.121.130:33089/ using document root <none> and password file <none>
I20260928 22:53:02.243808 4304 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:02.243860 4304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:02.244051 4304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:02.245213 4304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-1/data/instance:
uuid: "4f2f894f9aec4b5ba7577d8e3e295dfb"
format_stamp: "Formatted at 2026-09-28 22:53:02 on dist-test-slave-56vw"
server_key: "41039ba211551f2082ae9292d7155c76"
server_key_iv: "96fabcf58dfe30f1fac277ff8531fe27"
server_key_version: "encryptionkey@0"
I20260928 22:53:02.245576 4304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance:
uuid: "4f2f894f9aec4b5ba7577d8e3e295dfb"
format_stamp: "Formatted at 2026-09-28 22:53:02 on dist-test-slave-56vw"
server_key: "41039ba211551f2082ae9292d7155c76"
server_key_iv: "96fabcf58dfe30f1fac277ff8531fe27"
server_key_version: "encryptionkey@0"
I20260928 22:53:02.246987 4304 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 22:53:02.248271 4318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:02.248540 4304 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:53:02.248670 4304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-1/wal
uuid: "4f2f894f9aec4b5ba7577d8e3e295dfb"
format_stamp: "Formatted at 2026-09-28 22:53:02 on dist-test-slave-56vw"
server_key: "41039ba211551f2082ae9292d7155c76"
server_key_iv: "96fabcf58dfe30f1fac277ff8531fe27"
server_key_version: "encryptionkey@0"
I20260928 22:53:02.248764 4304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:02.268786 4304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:02.269188 4304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:02.269371 4304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:02.269658 4304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:53:02.270138 4304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:53:02.270191 4304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:02.270236 4304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:53:02.270267 4304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:02.276996 4304 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.130:36011
I20260928 22:53:02.277042 4431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.130:36011 every 8 connection(s)
I20260928 22:53:02.277437 4304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-1/data/info.pb
I20260928 22:53:02.281498 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 4304
I20260928 22:53:02.281615 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-1/wal/instance
I20260928 22:53:02.281818 16870 external_mini_cluster.cc:1428] Setting key 6b29b1883b7f350aa884b8b8fd3f765c
I20260928 22:53:02.282966 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.121.131:0
--local_ip_for_outbound_sockets=127.16.121.131
--webserver_interface=127.16.121.131
--webserver_port=0
--tserver_master_addrs=127.16.121.190:38769
--builtin_ntp_servers=127.16.121.148:45863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 22:53:02.283411 4152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:02.278251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.130:33797 (local address 127.16.121.190:38769)
0928 22:53:02.278346 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:02.278348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:02.278630 (+ 282us) server_negotiation.cc:407] Connection header received
0928 22:53:02.279238 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:02.279239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:02.279250 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:02.279307 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:02.279994 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:02.280489 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:02.281652 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:02.281892 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:02.281971 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:02.283025 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:02.283027 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:02.283027 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:53:02.283033 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:02.283078 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:53:02.283084 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:02.283151 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:02.283294 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 22:53:02.283330 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 22:53:02.283665 4434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:02.278187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:38769 (local address 127.16.121.130:33797)
0928 22:53:02.278386 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:02.278422 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:02.279104 (+ 682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:02.279365 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:02.279368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:02.279377 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:02.279902 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:02.279907 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:02.280592 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:02.280593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:02.281537 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:02.281540 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:02.281983 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:02.281984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:02.282156 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:02.282828 (+ 672us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:53:02.282829 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:02.282869 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:53:02.282902 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:02.283183 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:02.283189 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:02.283193 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:53:02.283290 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 22:53:02.283402 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":93,"threads_started":1}
I20260928 22:53:02.284134 4432 heartbeater.cc:347] Connected to a master server at 127.16.121.190:38769
I20260928 22:53:02.284248 4432 heartbeater.cc:464] Registering TS with master...
I20260928 22:53:02.284471 4432 heartbeater.cc:511] Master 127.16.121.190:38769 requested a full tablet report, sending...
I20260928 22:53:02.284929 4113 ts_manager.cc:194] Registered new tserver with Master: 4f2f894f9aec4b5ba7577d8e3e295dfb (127.16.121.130:36011)
I20260928 22:53:02.285321 4113 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.121.130:33797
W20260928 22:53:02.368363 4435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:02.368587 4435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:02.368611 4435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:02.369817 4435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:02.369863 4435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:02.369889 4435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.121.131
I20260928 22:53:02.371419 4435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:45863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.121.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.121.131
--webserver_port=0
--tserver_master_addrs=127.16.121.190:38769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.121.131
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:02.371702 4435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:02.372030 4435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:02.377365 4443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:02.377373 4440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:02.377369 4441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:02.377740 4435 server_base.cc:1034] running on GCE node
I20260928 22:53:02.377976 4435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:02.378265 4435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:02.379418 4435 hybrid_clock.cc:648] HybridClock initialized: now 1790635982379397 us; error 36 us; skew 500 ppm
I20260928 22:53:02.380810 4435 webserver.cc:460] Webserver started at http://127.16.121.131:39809/ using document root <none> and password file <none>
I20260928 22:53:02.381076 4435 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:02.381150 4435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:02.381302 4435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:02.382424 4435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-2/data/instance:
uuid: "c65c29bc220d46268493a92aa9aeeb5b"
format_stamp: "Formatted at 2026-09-28 22:53:02 on dist-test-slave-56vw"
server_key: "5d8c4fe6dbd6eba1446dc55e80985f01"
server_key_iv: "b83733fcebedcee3cde55ef502ebc9fa"
server_key_version: "encryptionkey@0"
I20260928 22:53:02.382795 4435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance:
uuid: "c65c29bc220d46268493a92aa9aeeb5b"
format_stamp: "Formatted at 2026-09-28 22:53:02 on dist-test-slave-56vw"
server_key: "5d8c4fe6dbd6eba1446dc55e80985f01"
server_key_iv: "b83733fcebedcee3cde55ef502ebc9fa"
server_key_version: "encryptionkey@0"
I20260928 22:53:02.384325 4435 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:53:02.385308 4449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:02.385697 4435 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:53:02.385828 4435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-2/wal
uuid: "c65c29bc220d46268493a92aa9aeeb5b"
format_stamp: "Formatted at 2026-09-28 22:53:02 on dist-test-slave-56vw"
server_key: "5d8c4fe6dbd6eba1446dc55e80985f01"
server_key_iv: "b83733fcebedcee3cde55ef502ebc9fa"
server_key_version: "encryptionkey@0"
I20260928 22:53:02.385915 4435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:02.396353 4435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:02.396699 4435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:02.396854 4435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:02.397089 4435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 22:53:02.397418 4435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 22:53:02.397481 4435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:02.397526 4435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 22:53:02.397553 4435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:02.403900 4435 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.131:34817
I20260928 22:53:02.403954 4562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.131:34817 every 8 connection(s)
I20260928 22:53:02.404284 4435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-2/data/info.pb
I20260928 22:53:02.409169 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 4435
I20260928 22:53:02.409301 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-2/wal/instance
I20260928 22:53:02.409507 16870 external_mini_cluster.cc:1428] Setting key 77a665ccf1fcc18b6e47ef74aab2752b
I20260928 22:53:02.411032 4152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:02.405093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.131:50259 (local address 127.16.121.190:38769)
0928 22:53:02.405192 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:02.405194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:02.405581 (+ 387us) server_negotiation.cc:407] Connection header received
0928 22:53:02.406118 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:02.406119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:02.406130 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:02.406191 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:02.407031 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:02.407741 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:02.409074 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:02.409414 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:02.409509 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:02.410731 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:02.410732 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:02.410732 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:53:02.410738 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:02.410754 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:53:02.410758 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:02.410832 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:02.410919 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 22:53:02.410952 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 22:53:02.411321 4565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:02.404992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:38769 (local address 127.16.121.131:50259)
0928 22:53:02.405364 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:02.405405 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:02.406025 (+ 620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:02.406270 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:02.406274 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:02.406283 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:02.406928 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:02.406936 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:02.407900 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:02.407901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:02.408956 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:02.408959 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:02.409586 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:02.409587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:02.409780 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:02.410551 (+ 771us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 22:53:02.410552 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:02.410582 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 22:53:02.410610 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:02.410833 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:02.410840 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:02.410843 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:53:02.410922 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 22:53:02.411104 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":112,"threads_started":1}
I20260928 22:53:02.411929 4563 heartbeater.cc:347] Connected to a master server at 127.16.121.190:38769
I20260928 22:53:02.412066 4563 heartbeater.cc:464] Registering TS with master...
I20260928 22:53:02.412284 4563 heartbeater.cc:511] Master 127.16.121.190:38769 requested a full tablet report, sending...
I20260928 22:53:02.412765 4113 ts_manager.cc:194] Registered new tserver with Master: c65c29bc220d46268493a92aa9aeeb5b (127.16.121.131:34817)
I20260928 22:53:02.413174 4113 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.121.131:50259
I20260928 22:53:02.421775 16870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 22:53:02.426518 4152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:02.423373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57422 (local address 127.16.121.190:38769)
0928 22:53:02.423496 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:02.423498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:02.423551 (+ 53us) server_negotiation.cc:407] Connection header received
0928 22:53:02.423622 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:02.423623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:02.423635 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:02.423692 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:02.423877 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:02.424440 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:02.425258 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:02.425471 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:02.425536 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:02.426210 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:02.426211 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:02.426211 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:53:02.426216 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:02.426231 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:53:02.426235 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:02.426288 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:02.426351 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 22:53:02.426388 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260928 22:53:02.428305 4113 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 22:53:02.429925 4113 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 22:53:02.436074 4571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:02.432259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.130:36011 (local address 127.0.0.1:42630)
0928 22:53:02.432385 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:02.432415 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:02.432502 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:02.432825 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:02.432828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:02.432831 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:53:02.432907 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:02.432909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:02.433864 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:02.433865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:02.434895 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:02.434898 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:02.435739 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:02.435740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:02.435821 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:02.435837 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 22:53:02.435914 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 22:53:02.435967 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":86,"threads_started":1}
I20260928 22:53:02.436082 4572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:02.432367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42630 (local address 127.16.121.130:36011)
0928 22:53:02.432563 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:02.432569 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:02.432578 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:53:02.432614 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:02.432616 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:02.432630 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:02.432726 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:53:02.433015 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:02.433773 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:02.434993 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:02.435653 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:02.435730 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:02.435818 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:02.435939 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 22:53:02.435994 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":85,"threads_started":1}
I20260928 22:53:02.436890 4573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:02.432483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.131:34817 (local address 127.0.0.1:35284)
0928 22:53:02.432568 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:02.432579 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:02.432630 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:02.433214 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:02.433215 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:02.433217 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:53:02.433274 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:02.433276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:02.434173 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:02.434174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:02.435356 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:02.435359 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:02.436668 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:02.436669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:02.436732 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:02.436735 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 22:53:02.436778 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 22:53:02.436822 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71,"thread_start_us":113,"threads_started":1}
I20260928 22:53:02.436920 4576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:02.432907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35284 (local address 127.16.121.131:34817)
0928 22:53:02.433049 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:02.433054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:02.433064 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:53:02.433108 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:02.433109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:02.433121 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:02.433215 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:53:02.433331 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:02.434086 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:02.435452 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:02.436579 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:02.436670 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:02.436780 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:02.436804 (+ 24us) server_negotiation.cc:299] Negotiation successful
0928 22:53:02.436840 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":71,"threads_started":1}
I20260928 22:53:02.436947 4366 tablet_service.cc:1467] Processing CreateTablet for tablet 7bbb2f0f400045f8af7ae7643cd427de (DEFAULT_TABLE table=test-table [id=ca350ebb922e414bb59bb27bf0a6086f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:53:02.437340 4366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bbb2f0f400045f8af7ae7643cd427de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:02.437770 4574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:02.432713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:42945 (local address 127.0.0.1:54364)
0928 22:53:02.432929 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:02.432938 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:02.433934 (+ 996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:02.434131 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:02.434132 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:02.434135 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:53:02.434179 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:02.434181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:02.435470 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:02.435470 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:02.436700 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:02.436703 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:02.437606 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:02.437606 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:02.437653 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:02.437657 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:53:02.437695 (+ 38us) client_negotiation.cc:240] Negotiation successful
0928 22:53:02.437725 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":101,"threads_started":1}
I20260928 22:53:02.437826 4575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:02.432838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54364 (local address 127.16.121.129:42945)
0928 22:53:02.433067 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:02.433071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:02.433081 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:53:02.434035 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:02.434038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:02.434054 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:02.434121 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:53:02.434260 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:02.435387 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:02.436794 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:02.437511 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:02.437599 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:02.437678 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:02.437705 (+ 27us) server_negotiation.cc:299] Negotiation successful
0928 22:53:02.437748 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":80,"threads_started":1}
I20260928 22:53:02.439772 4235 tablet_service.cc:1467] Processing CreateTablet for tablet 7bbb2f0f400045f8af7ae7643cd427de (DEFAULT_TABLE table=test-table [id=ca350ebb922e414bb59bb27bf0a6086f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:53:02.439824 4497 tablet_service.cc:1467] Processing CreateTablet for tablet 7bbb2f0f400045f8af7ae7643cd427de (DEFAULT_TABLE table=test-table [id=ca350ebb922e414bb59bb27bf0a6086f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 22:53:02.440119 4235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bbb2f0f400045f8af7ae7643cd427de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:02.440164 4497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bbb2f0f400045f8af7ae7643cd427de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:02.440726 4577 tablet_bootstrap.cc:492] T 7bbb2f0f400045f8af7ae7643cd427de P 4f2f894f9aec4b5ba7577d8e3e295dfb: Bootstrap starting.
I20260928 22:53:02.441457 4577 tablet_bootstrap.cc:654] T 7bbb2f0f400045f8af7ae7643cd427de P 4f2f894f9aec4b5ba7577d8e3e295dfb: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:02.441835 4577 log.cc:824] T 7bbb2f0f400045f8af7ae7643cd427de P 4f2f894f9aec4b5ba7577d8e3e295dfb: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:02.442689 4577 tablet_bootstrap.cc:492] T 7bbb2f0f400045f8af7ae7643cd427de P 4f2f894f9aec4b5ba7577d8e3e295dfb: No bootstrap required, opened a new log
I20260928 22:53:02.442763 4577 ts_tablet_manager.cc:1397] T 7bbb2f0f400045f8af7ae7643cd427de P 4f2f894f9aec4b5ba7577d8e3e295dfb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:53:02.443499 4579 tablet_bootstrap.cc:492] T 7bbb2f0f400045f8af7ae7643cd427de P c65c29bc220d46268493a92aa9aeeb5b: Bootstrap starting.
I20260928 22:53:02.443924 4580 tablet_bootstrap.cc:492] T 7bbb2f0f400045f8af7ae7643cd427de P 4941053186314a97b42634746ed3dc32: Bootstrap starting.
I20260928 22:53:02.444200 4579 tablet_bootstrap.cc:654] T 7bbb2f0f400045f8af7ae7643cd427de P c65c29bc220d46268493a92aa9aeeb5b: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:02.444485 4579 log.cc:824] T 7bbb2f0f400045f8af7ae7643cd427de P c65c29bc220d46268493a92aa9aeeb5b: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:02.444600 4580 tablet_bootstrap.cc:654] T 7bbb2f0f400045f8af7ae7643cd427de P 4941053186314a97b42634746ed3dc32: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:02.444954 4580 log.cc:824] T 7bbb2f0f400045f8af7ae7643cd427de P 4941053186314a97b42634746ed3dc32: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:02.445565 4579 tablet_bootstrap.cc:492] T 7bbb2f0f400045f8af7ae7643cd427de P c65c29bc220d46268493a92aa9aeeb5b: No bootstrap required, opened a new log
I20260928 22:53:02.445652 4579 ts_tablet_manager.cc:1397] T 7bbb2f0f400045f8af7ae7643cd427de P c65c29bc220d46268493a92aa9aeeb5b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:53:02.445750 4580 tablet_bootstrap.cc:492] T 7bbb2f0f400045f8af7ae7643cd427de P 4941053186314a97b42634746ed3dc32: No bootstrap required, opened a new log
I20260928 22:53:02.445657 4577 raft_consensus.cc:348] T 7bbb2f0f400045f8af7ae7643cd427de P 4f2f894f9aec4b5ba7577d8e3e295dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2f894f9aec4b5ba7577d8e3e295dfb" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 36011 } } peers { permanent_uuid: "c65c29bc220d46268493a92aa9aeeb5b" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34817 } } peers { permanent_uuid: "4941053186314a97b42634746ed3dc32" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 42945 } }
I20260928 22:53:02.445820 4580 ts_tablet_manager.cc:1397] T 7bbb2f0f400045f8af7ae7643cd427de P 4941053186314a97b42634746ed3dc32: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 22:53:02.445827 4577 raft_consensus.cc:374] T 7bbb2f0f400045f8af7ae7643cd427de P 4f2f894f9aec4b5ba7577d8e3e295dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:02.445863 4577 raft_consensus.cc:729] T 7bbb2f0f400045f8af7ae7643cd427de P 4f2f894f9aec4b5ba7577d8e3e295dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f2f894f9aec4b5ba7577d8e3e295dfb, State: Initialized, Role: FOLLOWER
I20260928 22:53:02.445997 4577 consensus_queue.cc:260] T 7bbb2f0f400045f8af7ae7643cd427de P 4f2f894f9aec4b5ba7577d8e3e295dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2f894f9aec4b5ba7577d8e3e295dfb" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 36011 } } peers { permanent_uuid: "c65c29bc220d46268493a92aa9aeeb5b" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34817 } } peers { permanent_uuid: "4941053186314a97b42634746ed3dc32" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 42945 } }
I20260928 22:53:02.446280 4577 ts_tablet_manager.cc:1428] T 7bbb2f0f400045f8af7ae7643cd427de P 4f2f894f9aec4b5ba7577d8e3e295dfb: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 22:53:02.446324 4432 heartbeater.cc:503] Master 127.16.121.190:38769 was elected leader, sending a full tablet report...
I20260928 22:53:02.448197 4579 raft_consensus.cc:348] T 7bbb2f0f400045f8af7ae7643cd427de P c65c29bc220d46268493a92aa9aeeb5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2f894f9aec4b5ba7577d8e3e295dfb" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 36011 } } peers { permanent_uuid: "c65c29bc220d46268493a92aa9aeeb5b" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34817 } } peers { permanent_uuid: "4941053186314a97b42634746ed3dc32" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 42945 } }
I20260928 22:53:02.448367 4579 raft_consensus.cc:374] T 7bbb2f0f400045f8af7ae7643cd427de P c65c29bc220d46268493a92aa9aeeb5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:02.448413 4579 raft_consensus.cc:729] T 7bbb2f0f400045f8af7ae7643cd427de P c65c29bc220d46268493a92aa9aeeb5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c65c29bc220d46268493a92aa9aeeb5b, State: Initialized, Role: FOLLOWER
I20260928 22:53:02.448541 4579 consensus_queue.cc:260] T 7bbb2f0f400045f8af7ae7643cd427de P c65c29bc220d46268493a92aa9aeeb5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2f894f9aec4b5ba7577d8e3e295dfb" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 36011 } } peers { permanent_uuid: "c65c29bc220d46268493a92aa9aeeb5b" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34817 } } peers { permanent_uuid: "4941053186314a97b42634746ed3dc32" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 42945 } }
I20260928 22:53:02.448568 4580 raft_consensus.cc:348] T 7bbb2f0f400045f8af7ae7643cd427de P 4941053186314a97b42634746ed3dc32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2f894f9aec4b5ba7577d8e3e295dfb" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 36011 } } peers { permanent_uuid: "c65c29bc220d46268493a92aa9aeeb5b" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34817 } } peers { permanent_uuid: "4941053186314a97b42634746ed3dc32" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 42945 } }
I20260928 22:53:02.448735 4580 raft_consensus.cc:374] T 7bbb2f0f400045f8af7ae7643cd427de P 4941053186314a97b42634746ed3dc32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:02.448769 4580 raft_consensus.cc:729] T 7bbb2f0f400045f8af7ae7643cd427de P 4941053186314a97b42634746ed3dc32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4941053186314a97b42634746ed3dc32, State: Initialized, Role: FOLLOWER
I20260928 22:53:02.448863 4579 ts_tablet_manager.cc:1428] T 7bbb2f0f400045f8af7ae7643cd427de P c65c29bc220d46268493a92aa9aeeb5b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 22:53:02.448922 4563 heartbeater.cc:503] Master 127.16.121.190:38769 was elected leader, sending a full tablet report...
I20260928 22:53:02.448923 4580 consensus_queue.cc:260] T 7bbb2f0f400045f8af7ae7643cd427de P 4941053186314a97b42634746ed3dc32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2f894f9aec4b5ba7577d8e3e295dfb" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 36011 } } peers { permanent_uuid: "c65c29bc220d46268493a92aa9aeeb5b" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34817 } } peers { permanent_uuid: "4941053186314a97b42634746ed3dc32" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 42945 } }
I20260928 22:53:02.449234 4301 heartbeater.cc:503] Master 127.16.121.190:38769 was elected leader, sending a full tablet report...
I20260928 22:53:02.449507 4580 ts_tablet_manager.cc:1428] T 7bbb2f0f400045f8af7ae7643cd427de P 4941053186314a97b42634746ed3dc32: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 22:53:02.491312 4585 raft_consensus.cc:482] T 7bbb2f0f400045f8af7ae7643cd427de P 4941053186314a97b42634746ed3dc32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:53:02.491464 4585 raft_consensus.cc:504] T 7bbb2f0f400045f8af7ae7643cd427de P 4941053186314a97b42634746ed3dc32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2f894f9aec4b5ba7577d8e3e295dfb" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 36011 } } peers { permanent_uuid: "c65c29bc220d46268493a92aa9aeeb5b" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34817 } } peers { permanent_uuid: "4941053186314a97b42634746ed3dc32" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 42945 } }
I20260928 22:53:02.491959 4585 leader_election.cc:288] T 7bbb2f0f400045f8af7ae7643cd427de P 4941053186314a97b42634746ed3dc32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f2f894f9aec4b5ba7577d8e3e295dfb (127.16.121.130:36011), c65c29bc220d46268493a92aa9aeeb5b (127.16.121.131:34817)
I20260928 22:53:02.495174 4303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:02.492207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.130:36011 (local address 127.16.121.129:36093)
0928 22:53:02.492248 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:02.492261 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:02.492306 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:02.492550 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:02.492552 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:02.492554 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:53:02.492632 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:02.492634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:02.493441 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:02.493443 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:02.494253 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:02.494256 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:02.494848 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:02.494850 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:02.494934 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:02.494939 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 22:53:02.494995 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 22:53:02.495036 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260928 22:53:02.495165 4572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:02.492285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:36093 (local address 127.16.121.130:36011)
0928 22:53:02.492337 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:02.492339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:02.492348 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:53:02.492378 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:02.492378 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:02.492388 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:02.492443 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:53:02.492729 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:02.493275 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:02.494347 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:02.494754 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:02.494812 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:02.494820 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:02.495011 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 22:53:02.495079 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 22:53:02.495276 4576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:02.492446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.121.129:45495 (local address 127.16.121.131:34817)
0928 22:53:02.492510 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:02.492512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:02.492548 (+ 36us) server_negotiation.cc:407] Connection header received
0928 22:53:02.492621 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:02.492622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:02.492632 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:02.492685 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 22:53:02.492935 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:02.493463 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:02.494444 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:02.494859 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:02.494936 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:02.494944 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:02.495154 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 22:53:02.495201 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 22:53:02.495324 4586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:02.492313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.131:34817 (local address 127.16.121.129:45495)
0928 22:53:02.492495 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:02.492505 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:02.492554 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:02.492725 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:02.492726 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:02.492728 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 22:53:02.492797 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:02.492799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:02.493562 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:02.493562 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:02.494340 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:02.494342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:02.494946 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:02.494947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:02.495077 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:02.495081 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 22:53:02.495134 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 22:53:02.495170 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":101,"threads_started":1}
I20260928 22:53:02.495563 4386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bbb2f0f400045f8af7ae7643cd427de" candidate_uuid: "4941053186314a97b42634746ed3dc32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f2f894f9aec4b5ba7577d8e3e295dfb" is_pre_election: true
I20260928 22:53:02.495657 4517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bbb2f0f400045f8af7ae7643cd427de" candidate_uuid: "4941053186314a97b42634746ed3dc32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c65c29bc220d46268493a92aa9aeeb5b" is_pre_election: true
I20260928 22:53:02.495743 4386 raft_consensus.cc:2454] T 7bbb2f0f400045f8af7ae7643cd427de P 4f2f894f9aec4b5ba7577d8e3e295dfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4941053186314a97b42634746ed3dc32 in term 0.
I20260928 22:53:02.495779 4517 raft_consensus.cc:2454] T 7bbb2f0f400045f8af7ae7643cd427de P c65c29bc220d46268493a92aa9aeeb5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4941053186314a97b42634746ed3dc32 in term 0.
I20260928 22:53:02.496025 4190 leader_election.cc:302] T 7bbb2f0f400045f8af7ae7643cd427de P 4941053186314a97b42634746ed3dc32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4941053186314a97b42634746ed3dc32, c65c29bc220d46268493a92aa9aeeb5b; no voters:
I20260928 22:53:02.496187 4585 raft_consensus.cc:2781] T 7bbb2f0f400045f8af7ae7643cd427de P 4941053186314a97b42634746ed3dc32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 22:53:02.496275 4585 raft_consensus.cc:482] T 7bbb2f0f400045f8af7ae7643cd427de P 4941053186314a97b42634746ed3dc32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 22:53:02.496311 4585 raft_consensus.cc:3037] T 7bbb2f0f400045f8af7ae7643cd427de P 4941053186314a97b42634746ed3dc32 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:02.497051 4585 raft_consensus.cc:504] T 7bbb2f0f400045f8af7ae7643cd427de P 4941053186314a97b42634746ed3dc32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2f894f9aec4b5ba7577d8e3e295dfb" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 36011 } } peers { permanent_uuid: "c65c29bc220d46268493a92aa9aeeb5b" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34817 } } peers { permanent_uuid: "4941053186314a97b42634746ed3dc32" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 42945 } }
I20260928 22:53:02.497205 4585 leader_election.cc:288] T 7bbb2f0f400045f8af7ae7643cd427de P 4941053186314a97b42634746ed3dc32 [CANDIDATE]: Term 1 election: Requested vote from peers 4f2f894f9aec4b5ba7577d8e3e295dfb (127.16.121.130:36011), c65c29bc220d46268493a92aa9aeeb5b (127.16.121.131:34817)
I20260928 22:53:02.497411 4386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bbb2f0f400045f8af7ae7643cd427de" candidate_uuid: "4941053186314a97b42634746ed3dc32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f2f894f9aec4b5ba7577d8e3e295dfb"
I20260928 22:53:02.497511 4386 raft_consensus.cc:3037] T 7bbb2f0f400045f8af7ae7643cd427de P 4f2f894f9aec4b5ba7577d8e3e295dfb [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:02.497587 4517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bbb2f0f400045f8af7ae7643cd427de" candidate_uuid: "4941053186314a97b42634746ed3dc32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c65c29bc220d46268493a92aa9aeeb5b"
I20260928 22:53:02.497713 4517 raft_consensus.cc:3037] T 7bbb2f0f400045f8af7ae7643cd427de P c65c29bc220d46268493a92aa9aeeb5b [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:02.498358 4386 raft_consensus.cc:2454] T 7bbb2f0f400045f8af7ae7643cd427de P 4f2f894f9aec4b5ba7577d8e3e295dfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4941053186314a97b42634746ed3dc32 in term 1.
I20260928 22:53:02.498548 4190 leader_election.cc:302] T 7bbb2f0f400045f8af7ae7643cd427de P 4941053186314a97b42634746ed3dc32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4941053186314a97b42634746ed3dc32, 4f2f894f9aec4b5ba7577d8e3e295dfb; no voters:
I20260928 22:53:02.498648 4517 raft_consensus.cc:2454] T 7bbb2f0f400045f8af7ae7643cd427de P c65c29bc220d46268493a92aa9aeeb5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4941053186314a97b42634746ed3dc32 in term 1.
I20260928 22:53:02.498690 4585 raft_consensus.cc:2781] T 7bbb2f0f400045f8af7ae7643cd427de P 4941053186314a97b42634746ed3dc32 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:53:02.498853 4585 raft_consensus.cc:686] T 7bbb2f0f400045f8af7ae7643cd427de P 4941053186314a97b42634746ed3dc32 [term 1 LEADER]: Becoming Leader. State: Replica: 4941053186314a97b42634746ed3dc32, State: Running, Role: LEADER
I20260928 22:53:02.498998 4585 consensus_queue.cc:237] T 7bbb2f0f400045f8af7ae7643cd427de P 4941053186314a97b42634746ed3dc32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2f894f9aec4b5ba7577d8e3e295dfb" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 36011 } } peers { permanent_uuid: "c65c29bc220d46268493a92aa9aeeb5b" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34817 } } peers { permanent_uuid: "4941053186314a97b42634746ed3dc32" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 42945 } }
I20260928 22:53:02.500160 4113 catalog_manager.cc:5495] T 7bbb2f0f400045f8af7ae7643cd427de P 4941053186314a97b42634746ed3dc32 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4941053186314a97b42634746ed3dc32 (127.16.121.129). New cstate: current_term: 1 leader_uuid: "4941053186314a97b42634746ed3dc32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2f894f9aec4b5ba7577d8e3e295dfb" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 36011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c65c29bc220d46268493a92aa9aeeb5b" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4941053186314a97b42634746ed3dc32" member_type: VOTER last_known_addr { host: "127.16.121.129" port: 42945 } health_report { overall_health: HEALTHY } } }
I20260928 22:53:02.512778 4575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:02.509928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54376 (local address 127.16.121.129:42945)
0928 22:53:02.510008 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:02.510010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:02.510019 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:53:02.510063 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:02.510063 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:02.510082 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:02.510144 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:53:02.510277 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:02.510744 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:02.511830 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:02.512097 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:02.512171 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:02.512383 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:53:02.512542 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:53:02.512619 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:02.512660 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 22:53:02.512681 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 22:53:02.514915 4517 raft_consensus.cc:1261] T 7bbb2f0f400045f8af7ae7643cd427de P c65c29bc220d46268493a92aa9aeeb5b [term 1 FOLLOWER]: Refusing update from remote peer 4941053186314a97b42634746ed3dc32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:53:02.514943 4386 raft_consensus.cc:1261] T 7bbb2f0f400045f8af7ae7643cd427de P 4f2f894f9aec4b5ba7577d8e3e295dfb [term 1 FOLLOWER]: Refusing update from remote peer 4941053186314a97b42634746ed3dc32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:53:02.515389 4587 consensus_queue.cc:1035] T 7bbb2f0f400045f8af7ae7643cd427de P 4941053186314a97b42634746ed3dc32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c65c29bc220d46268493a92aa9aeeb5b" member_type: VOTER last_known_addr { host: "127.16.121.131" port: 34817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:53:02.515585 4585 consensus_queue.cc:1035] T 7bbb2f0f400045f8af7ae7643cd427de P 4941053186314a97b42634746ed3dc32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f2f894f9aec4b5ba7577d8e3e295dfb" member_type: VOTER last_known_addr { host: "127.16.121.130" port: 36011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:53:02.518442 4592 mvcc.cc:205] Tried to move back new op lower bound from 7334444984378015744 to 7334444984317227008. Current Snapshot: MvccSnapshot[applied={T|T < 7334444984378015744}]
I20260928 22:53:02.518847 4591 mvcc.cc:205] Tried to move back new op lower bound from 7334444984378015744 to 7334444984317227008. Current Snapshot: MvccSnapshot[applied={T|T < 7334444984378015744}]
I20260928 22:53:02.520390 4113 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:53:02.520491 4113 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 22:53:02.521327 4113 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 446aea124f984ab9be364232dbf7f202: Sending DeleteTablet for 3 replicas of tablet 7bbb2f0f400045f8af7ae7643cd427de
I20260928 22:53:02.521708 4366 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bbb2f0f400045f8af7ae7643cd427de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:53:02 UTC) from {username='slave'} at 127.0.0.1:42630
I20260928 22:53:02.521757 4497 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bbb2f0f400045f8af7ae7643cd427de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:53:02 UTC) from {username='slave'} at 127.0.0.1:35284
I20260928 22:53:02.521950 4599 tablet_replica.cc:330] T 7bbb2f0f400045f8af7ae7643cd427de P 4f2f894f9aec4b5ba7577d8e3e295dfb: stopping tablet replica
I20260928 22:53:02.522055 4599 raft_consensus.cc:2229] T 7bbb2f0f400045f8af7ae7643cd427de P 4f2f894f9aec4b5ba7577d8e3e295dfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:53:02.522168 4599 raft_consensus.cc:2258] T 7bbb2f0f400045f8af7ae7643cd427de P 4f2f894f9aec4b5ba7577d8e3e295dfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:53:02.522157 4235 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bbb2f0f400045f8af7ae7643cd427de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 22:53:02 UTC) from {username='slave'} at 127.0.0.1:54364
I20260928 22:53:02.522373 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 4173
I20260928 22:53:02.523911 4600 tablet_replica.cc:330] T 7bbb2f0f400045f8af7ae7643cd427de P c65c29bc220d46268493a92aa9aeeb5b: stopping tablet replica
I20260928 22:53:02.524029 4599 ts_tablet_manager.cc:1905] T 7bbb2f0f400045f8af7ae7643cd427de P 4f2f894f9aec4b5ba7577d8e3e295dfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:53:02.524156 4600 raft_consensus.cc:2229] T 7bbb2f0f400045f8af7ae7643cd427de P c65c29bc220d46268493a92aa9aeeb5b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 22:53:02.524323 4600 raft_consensus.cc:2258] T 7bbb2f0f400045f8af7ae7643cd427de P c65c29bc220d46268493a92aa9aeeb5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 22:53:02.524940 4600 ts_tablet_manager.cc:1905] T 7bbb2f0f400045f8af7ae7643cd427de P c65c29bc220d46268493a92aa9aeeb5b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 22:53:02.525879 4599 ts_tablet_manager.cc:1918] T 7bbb2f0f400045f8af7ae7643cd427de P 4f2f894f9aec4b5ba7577d8e3e295dfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:53:02.525959 4599 log.cc:1196] T 7bbb2f0f400045f8af7ae7643cd427de P 4f2f894f9aec4b5ba7577d8e3e295dfb: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-1/wal/wals/7bbb2f0f400045f8af7ae7643cd427de
I20260928 22:53:02.526193 4599 ts_tablet_manager.cc:1939] T 7bbb2f0f400045f8af7ae7643cd427de P 4f2f894f9aec4b5ba7577d8e3e295dfb: Deleting consensus metadata
I20260928 22:53:02.526572 4600 ts_tablet_manager.cc:1918] T 7bbb2f0f400045f8af7ae7643cd427de P c65c29bc220d46268493a92aa9aeeb5b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 22:53:02.526651 4600 log.cc:1196] T 7bbb2f0f400045f8af7ae7643cd427de P c65c29bc220d46268493a92aa9aeeb5b: Deleting WAL directory at /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790635733329745-16870-0/minicluster-data/ts-2/wal/wals/7bbb2f0f400045f8af7ae7643cd427de
I20260928 22:53:02.526686 4099 catalog_manager.cc:4841] TS 4f2f894f9aec4b5ba7577d8e3e295dfb (127.16.121.130:36011): tablet 7bbb2f0f400045f8af7ae7643cd427de (table test-table [id=ca350ebb922e414bb59bb27bf0a6086f]) successfully deleted
I20260928 22:53:02.526916 4600 ts_tablet_manager.cc:1939] T 7bbb2f0f400045f8af7ae7643cd427de P c65c29bc220d46268493a92aa9aeeb5b: Deleting consensus metadata
I20260928 22:53:02.527398 4099 catalog_manager.cc:4841] TS c65c29bc220d46268493a92aa9aeeb5b (127.16.121.131:34817): tablet 7bbb2f0f400045f8af7ae7643cd427de (table test-table [id=ca350ebb922e414bb59bb27bf0a6086f]) successfully deleted
W20260928 22:53:02.528286 4099 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.121.129:42945 (error 108)
I20260928 22:53:02.528789 4574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:02.528641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.129:42945 (local address 127.0.0.1:54392)
0928 22:53:02.528704 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:02.528751 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.129:42945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 22:53:02.528834 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 4304
W20260928 22:53:02.528947 4099 catalog_manager.cc:4584] TS 4941053186314a97b42634746ed3dc32 (127.16.121.129:42945): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7bbb2f0f400045f8af7ae7643cd427de: Network error: Client connection negotiation failed: client connection to 127.16.121.129:42945: connect: Connection refused (error 111)
I20260928 22:53:02.533892 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 4435
I20260928 22:53:02.538517 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 4082
2026-09-28T22:53:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (759 ms)
[----------] 6 tests from EncryptionPolicyTest (9199 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635733329745-16870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:53:02 dist-test-slave-56vw krb5kdc[4605](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:53:02 dist-test-slave-56vw krb5kdc[4605](info): set up 2 sockets
Sep 28 22:53:02 dist-test-slave-56vw krb5kdc[4605](info): commencing operation
krb5kdc: starting...
W20260928 22:53:03.587261 16870 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Sep 28 22:53:03 dist-test-slave-56vw krb5kdc[4605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:53:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:53:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635733329745-16870-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635733329745-16870-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 22:53:03.715034 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44889
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635733329745-16870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:03.803236 4618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:03.803655 4618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:03.803704 4618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:03.804926 4618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:53:03.804978 4618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:03.804993 4618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:03.805006 4618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:53:03.805018 4618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:53:03.806521 4618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44889
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635733329745-16870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:03.806732 4618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:03.806937 4618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:03.812283 4623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:03.812500 4618 server_base.cc:1034] running on GCE node
W20260928 22:53:03.812289 4626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:03.812285 4624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:03.812913 4618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:03.813187 4618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:03.814339 4618 hybrid_clock.cc:648] HybridClock initialized: now 1790635983814327 us; error 32 us; skew 500 ppm
Sep 28 22:53:03 dist-test-slave-56vw krb5kdc[4605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635983, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:03.816936 4618 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 22:53:03.817337 4618 webserver.cc:460] Webserver started at http://127.0.0.1:34005/ using document root <none> and password file <none>
I20260928 22:53:03.817624 4618 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:03.817675 4618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:03.817811 4618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:03.819438 4618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "4286b82420ec44c48a060897239ef08b"
format_stamp: "Formatted at 2026-09-28 22:53:03 on dist-test-slave-56vw"
server_key: "1840a49c76b7d6f13465e843224fa3cc"
server_key_iv: "ebbeba5c65652517f2e5a5e12dc3f957"
server_key_version: "encryptionkey@0"
I20260928 22:53:03.819957 4618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "4286b82420ec44c48a060897239ef08b"
format_stamp: "Formatted at 2026-09-28 22:53:03 on dist-test-slave-56vw"
server_key: "1840a49c76b7d6f13465e843224fa3cc"
server_key_iv: "ebbeba5c65652517f2e5a5e12dc3f957"
server_key_version: "encryptionkey@0"
I20260928 22:53:03.821305 4618 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:53:03.822166 4633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:03.822454 4618 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:53:03.822569 4618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "4286b82420ec44c48a060897239ef08b"
format_stamp: "Formatted at 2026-09-28 22:53:03 on dist-test-slave-56vw"
server_key: "1840a49c76b7d6f13465e843224fa3cc"
server_key_iv: "ebbeba5c65652517f2e5a5e12dc3f957"
server_key_version: "encryptionkey@0"
I20260928 22:53:03.822654 4618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:03.849738 4618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:03.852319 4618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:03.852507 4618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:03.856676 4618 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44889
I20260928 22:53:03.856730 4685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44889 every 8 connection(s)
I20260928 22:53:03.857121 4618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:53:03.858110 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:03.860520 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 4618
I20260928 22:53:03.860661 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:53:03.860857 16870 external_mini_cluster.cc:1428] Setting key 326a8eb65c9dfcdb1e4fc269086589e6
I20260928 22:53:03.861013 4686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4286b82420ec44c48a060897239ef08b: Bootstrap starting.
I20260928 22:53:03.862154 4686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4286b82420ec44c48a060897239ef08b: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:03.862748 4686 log.cc:824] T 00000000000000000000000000000000 P 4286b82420ec44c48a060897239ef08b: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:03.863772 4686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4286b82420ec44c48a060897239ef08b: No bootstrap required, opened a new log
I20260928 22:53:03.866144 4686 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4286b82420ec44c48a060897239ef08b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4286b82420ec44c48a060897239ef08b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44889 } }
I20260928 22:53:03.866333 4686 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4286b82420ec44c48a060897239ef08b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:03.866385 4686 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4286b82420ec44c48a060897239ef08b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4286b82420ec44c48a060897239ef08b, State: Initialized, Role: FOLLOWER
I20260928 22:53:03.866513 4686 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4286b82420ec44c48a060897239ef08b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4286b82420ec44c48a060897239ef08b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44889 } }
I20260928 22:53:03.866591 4686 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4286b82420ec44c48a060897239ef08b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:53:03.866623 4686 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4286b82420ec44c48a060897239ef08b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:53:03.866670 4686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4286b82420ec44c48a060897239ef08b [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:03.867494 4686 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4286b82420ec44c48a060897239ef08b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4286b82420ec44c48a060897239ef08b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44889 } }
I20260928 22:53:03.867619 4686 leader_election.cc:302] T 00000000000000000000000000000000 P 4286b82420ec44c48a060897239ef08b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4286b82420ec44c48a060897239ef08b; no voters:
I20260928 22:53:03.867842 4686 leader_election.cc:288] T 00000000000000000000000000000000 P 4286b82420ec44c48a060897239ef08b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:53:03.867893 4691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4286b82420ec44c48a060897239ef08b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:53:03.868076 4691 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4286b82420ec44c48a060897239ef08b [term 1 LEADER]: Becoming Leader. State: Replica: 4286b82420ec44c48a060897239ef08b, State: Running, Role: LEADER
I20260928 22:53:03.868155 4686 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4286b82420ec44c48a060897239ef08b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:53:03.868214 4691 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4286b82420ec44c48a060897239ef08b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4286b82420ec44c48a060897239ef08b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44889 } }
I20260928 22:53:03.868583 4691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4286b82420ec44c48a060897239ef08b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4286b82420ec44c48a060897239ef08b. Latest consensus state: current_term: 1 leader_uuid: "4286b82420ec44c48a060897239ef08b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4286b82420ec44c48a060897239ef08b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44889 } } }
I20260928 22:53:03.868677 4691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4286b82420ec44c48a060897239ef08b [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:03.868590 4692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4286b82420ec44c48a060897239ef08b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4286b82420ec44c48a060897239ef08b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4286b82420ec44c48a060897239ef08b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44889 } } }
I20260928 22:53:03.868884 4692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4286b82420ec44c48a060897239ef08b [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:03.868906 4698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:53:03.869553 4698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 22:53:03 dist-test-slave-56vw krb5kdc[4605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 22:53:03.871538 4698 catalog_manager.cc:1347] Generated new cluster ID: b568a61e316848e48f97518acdb7c479
I20260928 22:53:03.871592 4698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:53:03.874586 4688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:03.861497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34159 (local address 127.0.0.1:44889)
0928 22:53:03.861760 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:03.861769 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:03.861824 (+ 55us) server_negotiation.cc:407] Connection header received
0928 22:53:03.862470 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:03.862479 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:03.862537 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:03.862699 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:03.863699 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:03.865126 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:03.865984 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:03.866625 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:03.866741 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:03.870213 (+ 3472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:03.870217 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:03.870217 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:03.870231 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:03.872759 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:03.873244 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:03.873247 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:03.873248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:03.873365 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:03.873609 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:03.873610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:03.873610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:03.873944 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:03.874157 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:03.874251 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 22:53:03.874358 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":98,"threads_started":1}
I20260928 22:53:03.875666 16870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:53:03.884773 4688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:03.877229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33197 (local address 127.0.0.1:44889)
0928 22:53:03.877273 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:03.877276 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:03.877457 (+ 181us) server_negotiation.cc:407] Connection header received
0928 22:53:03.877531 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:03.877532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:03.877541 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:03.877587 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:03.878290 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:03.879077 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:03.880077 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:03.880399 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:03.880435 (+ 36us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:03.881863 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:03.881865 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:03.881866 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:03.881873 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:03.883591 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:03.883984 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:03.883985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:03.883985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:03.884034 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:03.884217 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:03.884217 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:03.884217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:03.884331 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:03.884416 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:03.884619 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 22:53:03.884676 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:53:03.893718 4698 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:53:03.894481 4698 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:53:03.900328 4698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4286b82420ec44c48a060897239ef08b: Generated new TSK 0
I20260928 22:53:03.900635 4698 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:53:03.911607 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 4618
W20260928 22:53:03.916164 4603 connection.cc:535] client connection to 127.0.0.1:44889 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44889): Connection reset by peer (error 104)
2026-09-28T22:53:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1371 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635733329745-16870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:53:03 dist-test-slave-56vw krb5kdc[4717](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:53:03 dist-test-slave-56vw krb5kdc[4717](info): set up 2 sockets
Sep 28 22:53:03 dist-test-slave-56vw krb5kdc[4717](info): commencing operation
krb5kdc: starting...
W20260928 22:53:04.967171 16870 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Sep 28 22:53:05 dist-test-slave-56vw krb5kdc[4717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:53:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:53:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635733329745-16870-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635733329745-16870-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 22:53:05.093890 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40931
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635733329745-16870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:05.176028 4730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:05.176245 4730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:05.176270 4730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:05.177547 4730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:53:05.177596 4730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:05.177613 4730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:05.177625 4730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:53:05.177673 4730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:53:05.179286 4730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40931
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635733329745-16870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:05.179556 4730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:05.179813 4730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:05.184854 4735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:05.184954 4736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:05.184844 4738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:05.185099 4730 server_base.cc:1034] running on GCE node
I20260928 22:53:05.185405 4730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:05.185738 4730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:05.186875 4730 hybrid_clock.cc:648] HybridClock initialized: now 1790635985186874 us; error 38 us; skew 500 ppm
Sep 28 22:53:05 dist-test-slave-56vw krb5kdc[4717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:05.189566 4730 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 22:53:05.189987 4730 webserver.cc:460] Webserver started at http://127.0.0.1:42471/ using document root <none> and password file <none>
I20260928 22:53:05.190191 4730 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:05.190232 4730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:05.190322 4730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:05.191450 4730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "7a5a1881d46e49c89035514e48b8dee7"
format_stamp: "Formatted at 2026-09-28 22:53:05 on dist-test-slave-56vw"
server_key: "0db03392ccfe115d8d16831452d40059"
server_key_iv: "5badaf4fffca21679fb0fe641bf4eea6"
server_key_version: "encryptionkey@0"
I20260928 22:53:05.191756 4730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "7a5a1881d46e49c89035514e48b8dee7"
format_stamp: "Formatted at 2026-09-28 22:53:05 on dist-test-slave-56vw"
server_key: "0db03392ccfe115d8d16831452d40059"
server_key_iv: "5badaf4fffca21679fb0fe641bf4eea6"
server_key_version: "encryptionkey@0"
I20260928 22:53:05.193049 4730 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:53:05.193804 4745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:05.193970 4730 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:53:05.194077 4730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "7a5a1881d46e49c89035514e48b8dee7"
format_stamp: "Formatted at 2026-09-28 22:53:05 on dist-test-slave-56vw"
server_key: "0db03392ccfe115d8d16831452d40059"
server_key_iv: "5badaf4fffca21679fb0fe641bf4eea6"
server_key_version: "encryptionkey@0"
I20260928 22:53:05.194131 4730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:05.207218 4730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:05.209174 4730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:05.209287 4730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:05.213299 4730 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40931
I20260928 22:53:05.213356 4797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40931 every 8 connection(s)
I20260928 22:53:05.213722 4730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:53:05.214730 4798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:05.217128 4798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a5a1881d46e49c89035514e48b8dee7: Bootstrap starting.
I20260928 22:53:05.217741 4798 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a5a1881d46e49c89035514e48b8dee7: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:05.218035 4798 log.cc:824] T 00000000000000000000000000000000 P 7a5a1881d46e49c89035514e48b8dee7: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:05.218743 4798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a5a1881d46e49c89035514e48b8dee7: No bootstrap required, opened a new log
I20260928 22:53:05.219323 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 4730
I20260928 22:53:05.219413 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:53:05.219626 16870 external_mini_cluster.cc:1428] Setting key 279a19b8e6d43b77a73ca93e78fe2a73
I20260928 22:53:05.221233 4798 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a5a1881d46e49c89035514e48b8dee7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5a1881d46e49c89035514e48b8dee7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40931 } }
I20260928 22:53:05.221457 4798 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a5a1881d46e49c89035514e48b8dee7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:05.221494 4798 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a5a1881d46e49c89035514e48b8dee7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a5a1881d46e49c89035514e48b8dee7, State: Initialized, Role: FOLLOWER
I20260928 22:53:05.221626 4798 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a5a1881d46e49c89035514e48b8dee7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5a1881d46e49c89035514e48b8dee7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40931 } }
I20260928 22:53:05.221710 4798 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a5a1881d46e49c89035514e48b8dee7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:53:05.221741 4798 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a5a1881d46e49c89035514e48b8dee7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:53:05.221794 4798 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a5a1881d46e49c89035514e48b8dee7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:05.222354 4798 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a5a1881d46e49c89035514e48b8dee7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5a1881d46e49c89035514e48b8dee7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40931 } }
I20260928 22:53:05.222472 4798 leader_election.cc:302] T 00000000000000000000000000000000 P 7a5a1881d46e49c89035514e48b8dee7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a5a1881d46e49c89035514e48b8dee7; no voters:
I20260928 22:53:05.222664 4798 leader_election.cc:288] T 00000000000000000000000000000000 P 7a5a1881d46e49c89035514e48b8dee7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:53:05.222693 4803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a5a1881d46e49c89035514e48b8dee7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:53:05.222800 4803 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a5a1881d46e49c89035514e48b8dee7 [term 1 LEADER]: Becoming Leader. State: Replica: 7a5a1881d46e49c89035514e48b8dee7, State: Running, Role: LEADER
I20260928 22:53:05.222932 4798 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a5a1881d46e49c89035514e48b8dee7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:53:05.222950 4803 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a5a1881d46e49c89035514e48b8dee7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5a1881d46e49c89035514e48b8dee7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40931 } }
I20260928 22:53:05.223407 4803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a5a1881d46e49c89035514e48b8dee7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a5a1881d46e49c89035514e48b8dee7. Latest consensus state: current_term: 1 leader_uuid: "7a5a1881d46e49c89035514e48b8dee7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5a1881d46e49c89035514e48b8dee7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40931 } } }
I20260928 22:53:05.223650 4803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a5a1881d46e49c89035514e48b8dee7 [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:05.223570 4805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a5a1881d46e49c89035514e48b8dee7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a5a1881d46e49c89035514e48b8dee7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5a1881d46e49c89035514e48b8dee7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40931 } } }
I20260928 22:53:05.224114 4805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a5a1881d46e49c89035514e48b8dee7 [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:05.224241 4812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:53:05.224797 4812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:53:05.226387 4802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:05.220173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46073 (local address 127.0.0.1:40931)
0928 22:53:05.220405 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:05.220414 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:05.220453 (+ 39us) server_negotiation.cc:407] Connection header received
0928 22:53:05.221441 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:05.221450 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:05.221519 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:05.221673 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:05.221934 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:05.223031 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:05.224127 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:05.224727 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:05.224836 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:05.225644 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:05.225648 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:05.225648 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:53:05.225860 (+ 212us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:05.225898 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:53:05.225915 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:05.225992 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:05.226137 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 22:53:05.226163 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":89,"threads_started":1}
I20260928 22:53:05.227492 4812 catalog_manager.cc:1347] Generated new cluster ID: 450f0a6e0d7f4546881e6a5a981a9321
I20260928 22:53:05.227492 16870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:53:05.227577 4812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:53:05.243319 4802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:05.239476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35883 (local address 127.0.0.1:40931)
0928 22:53:05.239555 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:05.239557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:05.239700 (+ 143us) server_negotiation.cc:407] Connection header received
0928 22:53:05.239775 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:05.239776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:05.239789 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:05.239865 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:05.240025 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:05.240747 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:05.241865 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:05.242164 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:05.242227 (+ 63us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:05.243025 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:05.243027 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:05.243027 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:53:05.243033 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:05.243070 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:53:05.243075 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:05.243130 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:05.243216 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 22:53:05.243230 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 22:53:05.251783 4812 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:53:05.252344 4812 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:53:05.256273 4812 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a5a1881d46e49c89035514e48b8dee7: Generated new TSK 0
I20260928 22:53:05.256548 4812 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:53:05.265753 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 4730
W20260928 22:53:05.270247 4715 connection.cc:535] client connection to 127.0.0.1:40931 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40931): Connection reset by peer (error 104)
2026-09-28T22:53:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1354 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635733329745-16870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:53:05 dist-test-slave-56vw krb5kdc[4829](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:53:05 dist-test-slave-56vw krb5kdc[4829](info): set up 2 sockets
Sep 28 22:53:05 dist-test-slave-56vw krb5kdc[4829](info): commencing operation
krb5kdc: starting...
W20260928 22:53:06.319631 16870 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Sep 28 22:53:06 dist-test-slave-56vw krb5kdc[4829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:53:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:53:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635733329745-16870-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635733329745-16870-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 22:53:06.443368 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34923
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635733329745-16870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:06.528177 4842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:06.528414 4842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:06.528446 4842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:06.529770 4842 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:53:06.529820 4842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:06.529835 4842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:06.529858 4842 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:53:06.529871 4842 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:53:06.531529 4842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34923
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635733329745-16870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:06.531755 4842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:06.531942 4842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:06.537235 4847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:06.537235 4848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:06.537295 4850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:06.538007 4842 server_base.cc:1034] running on GCE node
I20260928 22:53:06.538322 4842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:06.538681 4842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:06.539882 4842 hybrid_clock.cc:648] HybridClock initialized: now 1790635986539852 us; error 43 us; skew 500 ppm
Sep 28 22:53:06 dist-test-slave-56vw krb5kdc[4829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:06.542635 4842 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 22:53:06.543208 4842 webserver.cc:460] Webserver started at http://127.0.0.1:39829/ using document root <none> and password file <none>
I20260928 22:53:06.543454 4842 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:06.543509 4842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:06.543645 4842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:06.544837 4842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "831578f3d1224a27858e2089e14986d8"
format_stamp: "Formatted at 2026-09-28 22:53:06 on dist-test-slave-56vw"
server_key: "abde70c4f26a657b942cf6bb83d7c717"
server_key_iv: "de9f6bd8d4a30522b36dc973a91c2176"
server_key_version: "encryptionkey@0"
I20260928 22:53:06.545225 4842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "831578f3d1224a27858e2089e14986d8"
format_stamp: "Formatted at 2026-09-28 22:53:06 on dist-test-slave-56vw"
server_key: "abde70c4f26a657b942cf6bb83d7c717"
server_key_iv: "de9f6bd8d4a30522b36dc973a91c2176"
server_key_version: "encryptionkey@0"
I20260928 22:53:06.546666 4842 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:53:06.547710 4857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:06.547967 4842 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 22:53:06.548077 4842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "831578f3d1224a27858e2089e14986d8"
format_stamp: "Formatted at 2026-09-28 22:53:06 on dist-test-slave-56vw"
server_key: "abde70c4f26a657b942cf6bb83d7c717"
server_key_iv: "de9f6bd8d4a30522b36dc973a91c2176"
server_key_version: "encryptionkey@0"
I20260928 22:53:06.548159 4842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:06.551625 4842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:06.553675 4842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:06.553818 4842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:06.557715 4842 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34923
I20260928 22:53:06.557745 4909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34923 every 8 connection(s)
I20260928 22:53:06.558105 4842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:53:06.558553 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 4842
I20260928 22:53:06.558640 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:53:06.558858 16870 external_mini_cluster.cc:1428] Setting key 81f45aeed8404f51be06dc91a9fded3d
I20260928 22:53:06.559216 4910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:06.561766 4913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:06.559398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33711 (local address 127.0.0.1:34923)
0928 22:53:06.559595 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:06.559612 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:06.559637 (+ 25us) server_negotiation.cc:407] Connection header received
0928 22:53:06.560309 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:06.560317 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:06.560347 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:06.560626 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:06.561327 (+ 701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:06.561331 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:06.561332 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:53:06.561336 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:06.561379 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:53:06.561389 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:06.561460 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:06.561568 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 22:53:06.561609 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":89,"threads_started":1}
I20260928 22:53:06.562549 4910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 831578f3d1224a27858e2089e14986d8: Bootstrap starting.
I20260928 22:53:06.562863 16870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:53:06.563289 4910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 831578f3d1224a27858e2089e14986d8: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:06.563591 4910 log.cc:824] T 00000000000000000000000000000000 P 831578f3d1224a27858e2089e14986d8: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:06.564368 4910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 831578f3d1224a27858e2089e14986d8: No bootstrap required, opened a new log
I20260928 22:53:06.565568 4913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:06.564392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35421 (local address 127.0.0.1:34923)
0928 22:53:06.564467 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:06.564470 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:06.564605 (+ 135us) server_negotiation.cc:407] Connection header received
0928 22:53:06.564660 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:06.564660 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:06.564665 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:06.564711 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:06.565292 (+ 581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:06.565293 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:06.565293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:53:06.565297 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:06.565309 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:53:06.565312 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:06.565355 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:06.565409 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 22:53:06.565419 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 22:53:06.566726 4910 raft_consensus.cc:348] T 00000000000000000000000000000000 P 831578f3d1224a27858e2089e14986d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "831578f3d1224a27858e2089e14986d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34923 } }
I20260928 22:53:06.566890 4910 raft_consensus.cc:374] T 00000000000000000000000000000000 P 831578f3d1224a27858e2089e14986d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:06.566933 4910 raft_consensus.cc:729] T 00000000000000000000000000000000 P 831578f3d1224a27858e2089e14986d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 831578f3d1224a27858e2089e14986d8, State: Initialized, Role: FOLLOWER
I20260928 22:53:06.567138 4910 consensus_queue.cc:260] T 00000000000000000000000000000000 P 831578f3d1224a27858e2089e14986d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "831578f3d1224a27858e2089e14986d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34923 } }
I20260928 22:53:06.567265 4920 raft_consensus.cc:482] T 00000000000000000000000000000000 P 831578f3d1224a27858e2089e14986d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 22:53:06.567328 4920 raft_consensus.cc:504] T 00000000000000000000000000000000 P 831578f3d1224a27858e2089e14986d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "831578f3d1224a27858e2089e14986d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34923 } }
I20260928 22:53:06.567246 4910 raft_consensus.cc:388] T 00000000000000000000000000000000 P 831578f3d1224a27858e2089e14986d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:53:06.567440 4910 raft_consensus.cc:482] T 00000000000000000000000000000000 P 831578f3d1224a27858e2089e14986d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:53:06.567425 4920 leader_election.cc:302] T 00000000000000000000000000000000 P 831578f3d1224a27858e2089e14986d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 831578f3d1224a27858e2089e14986d8; no voters:
I20260928 22:53:06.567476 4910 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 831578f3d1224a27858e2089e14986d8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:06.567598 4920 leader_election.cc:288] T 00000000000000000000000000000000 P 831578f3d1224a27858e2089e14986d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260928 22:53:06.568075 4910 raft_consensus.cc:504] T 00000000000000000000000000000000 P 831578f3d1224a27858e2089e14986d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "831578f3d1224a27858e2089e14986d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34923 } }
I20260928 22:53:06.568141 4910 leader_election.cc:302] T 00000000000000000000000000000000 P 831578f3d1224a27858e2089e14986d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 831578f3d1224a27858e2089e14986d8; no voters:
I20260928 22:53:06.568187 4921 raft_consensus.cc:2741] T 00000000000000000000000000000000 P 831578f3d1224a27858e2089e14986d8 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260928 22:53:06.568194 4910 leader_election.cc:288] T 00000000000000000000000000000000 P 831578f3d1224a27858e2089e14986d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:53:06.568356 4920 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 831578f3d1224a27858e2089e14986d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:53:06.568351 4921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 831578f3d1224a27858e2089e14986d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "831578f3d1224a27858e2089e14986d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34923 } } }
I20260928 22:53:06.568427 4920 raft_consensus.cc:686] T 00000000000000000000000000000000 P 831578f3d1224a27858e2089e14986d8 [term 1 LEADER]: Becoming Leader. State: Replica: 831578f3d1224a27858e2089e14986d8, State: Running, Role: LEADER
I20260928 22:53:06.568439 4921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 831578f3d1224a27858e2089e14986d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:53:06.568491 4910 sys_catalog.cc:564] T 00000000000000000000000000000000 P 831578f3d1224a27858e2089e14986d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:53:06.568491 4920 consensus_queue.cc:237] T 00000000000000000000000000000000 P 831578f3d1224a27858e2089e14986d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "831578f3d1224a27858e2089e14986d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34923 } }
I20260928 22:53:06.568825 4921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 831578f3d1224a27858e2089e14986d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 831578f3d1224a27858e2089e14986d8. Latest consensus state: current_term: 1 leader_uuid: "831578f3d1224a27858e2089e14986d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "831578f3d1224a27858e2089e14986d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34923 } } }
I20260928 22:53:06.568895 4921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 831578f3d1224a27858e2089e14986d8 [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:06.569517 4925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:53:06.570012 4925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:53:06.571569 4925 catalog_manager.cc:1347] Generated new cluster ID: 827f72bf70594b4487e921ddfdd47498
I20260928 22:53:06.571623 4925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:53:06.590551 4925 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:53:06.590747 4925 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:53:06.596644 4925 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 831578f3d1224a27858e2089e14986d8: Generated new TSK 0
I20260928 22:53:06.596877 4925 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:53:06.601377 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 4842
2026-09-28T22:53:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1335 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635733329745-16870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:53:06 dist-test-slave-56vw krb5kdc[4941](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:53:06 dist-test-slave-56vw krb5kdc[4941](info): set up 2 sockets
Sep 28 22:53:06 dist-test-slave-56vw krb5kdc[4941](info): commencing operation
krb5kdc: starting...
W20260928 22:53:07.655277 16870 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Sep 28 22:53:07 dist-test-slave-56vw krb5kdc[4941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:53:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:53:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635733329745-16870-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635733329745-16870-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 22:53:07.787506 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39141
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635733329745-16870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:07.871444 4954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:07.871748 4954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:07.871796 4954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:07.873217 4954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:53:07.873302 4954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:07.873330 4954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:07.873355 4954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:53:07.873389 4954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:53:07.875157 4954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635733329745-16870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:07.875488 4954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:07.875751 4954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:07.881428 4962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:07.881600 4959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:07.881623 4954 server_base.cc:1034] running on GCE node
W20260928 22:53:07.881428 4960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:07.882150 4954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:07.882429 4954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:07.883579 4954 hybrid_clock.cc:648] HybridClock initialized: now 1790635987883568 us; error 30 us; skew 500 ppm
Sep 28 22:53:07 dist-test-slave-56vw krb5kdc[4941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635987, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:07.886359 4954 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 22:53:07.886816 4954 webserver.cc:460] Webserver started at http://127.0.0.1:45691/ using document root <none> and password file <none>
I20260928 22:53:07.887104 4954 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:07.887187 4954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:07.887346 4954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:07.888468 4954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "075b57115a0143ae996773816dece3a9"
format_stamp: "Formatted at 2026-09-28 22:53:07 on dist-test-slave-56vw"
server_key: "0078e106b103e3b5ca6ca8c58802cc8e"
server_key_iv: "ca6bd546a6c5009b5ac03d6aad7c2e43"
server_key_version: "encryptionkey@0"
I20260928 22:53:07.888868 4954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "075b57115a0143ae996773816dece3a9"
format_stamp: "Formatted at 2026-09-28 22:53:07 on dist-test-slave-56vw"
server_key: "0078e106b103e3b5ca6ca8c58802cc8e"
server_key_iv: "ca6bd546a6c5009b5ac03d6aad7c2e43"
server_key_version: "encryptionkey@0"
I20260928 22:53:07.890276 4954 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:53:07.891170 4969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:07.891378 4954 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:53:07.891487 4954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "075b57115a0143ae996773816dece3a9"
format_stamp: "Formatted at 2026-09-28 22:53:07 on dist-test-slave-56vw"
server_key: "0078e106b103e3b5ca6ca8c58802cc8e"
server_key_iv: "ca6bd546a6c5009b5ac03d6aad7c2e43"
server_key_version: "encryptionkey@0"
I20260928 22:53:07.891582 4954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:07.906088 4954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:07.909106 4954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:07.909302 4954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:07.913623 4954 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39141
I20260928 22:53:07.913681 5021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39141 every 8 connection(s)
I20260928 22:53:07.914124 4954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:53:07.915347 5022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:07.917840 5022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 075b57115a0143ae996773816dece3a9: Bootstrap starting.
I20260928 22:53:07.918510 5022 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 075b57115a0143ae996773816dece3a9: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:07.918826 5022 log.cc:824] T 00000000000000000000000000000000 P 075b57115a0143ae996773816dece3a9: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:07.919669 5022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 075b57115a0143ae996773816dece3a9: No bootstrap required, opened a new log
I20260928 22:53:07.921403 5022 raft_consensus.cc:348] T 00000000000000000000000000000000 P 075b57115a0143ae996773816dece3a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "075b57115a0143ae996773816dece3a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39141 } }
I20260928 22:53:07.921602 5022 raft_consensus.cc:374] T 00000000000000000000000000000000 P 075b57115a0143ae996773816dece3a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:07.921658 5022 raft_consensus.cc:729] T 00000000000000000000000000000000 P 075b57115a0143ae996773816dece3a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075b57115a0143ae996773816dece3a9, State: Initialized, Role: FOLLOWER
I20260928 22:53:07.921849 5022 consensus_queue.cc:260] T 00000000000000000000000000000000 P 075b57115a0143ae996773816dece3a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "075b57115a0143ae996773816dece3a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39141 } }
I20260928 22:53:07.921942 5022 raft_consensus.cc:388] T 00000000000000000000000000000000 P 075b57115a0143ae996773816dece3a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:53:07.921988 5022 raft_consensus.cc:482] T 00000000000000000000000000000000 P 075b57115a0143ae996773816dece3a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:53:07.922050 5022 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 075b57115a0143ae996773816dece3a9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:07.922659 5022 raft_consensus.cc:504] T 00000000000000000000000000000000 P 075b57115a0143ae996773816dece3a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "075b57115a0143ae996773816dece3a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39141 } }
I20260928 22:53:07.922807 5022 leader_election.cc:302] T 00000000000000000000000000000000 P 075b57115a0143ae996773816dece3a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 075b57115a0143ae996773816dece3a9; no voters:
I20260928 22:53:07.922943 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 4954
I20260928 22:53:07.923069 5022 leader_election.cc:288] T 00000000000000000000000000000000 P 075b57115a0143ae996773816dece3a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:53:07.923091 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:53:07.923161 5025 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 075b57115a0143ae996773816dece3a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:53:07.923306 16870 external_mini_cluster.cc:1428] Setting key 2a52cb2c9b29c99fe04682efa228e6a4
I20260928 22:53:07.923352 5025 raft_consensus.cc:686] T 00000000000000000000000000000000 P 075b57115a0143ae996773816dece3a9 [term 1 LEADER]: Becoming Leader. State: Replica: 075b57115a0143ae996773816dece3a9, State: Running, Role: LEADER
I20260928 22:53:07.923377 5022 sys_catalog.cc:564] T 00000000000000000000000000000000 P 075b57115a0143ae996773816dece3a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:53:07.923455 5025 consensus_queue.cc:237] T 00000000000000000000000000000000 P 075b57115a0143ae996773816dece3a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "075b57115a0143ae996773816dece3a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39141 } }
I20260928 22:53:07.924125 5026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 075b57115a0143ae996773816dece3a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "075b57115a0143ae996773816dece3a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "075b57115a0143ae996773816dece3a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39141 } } }
I20260928 22:53:07.924239 5026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 075b57115a0143ae996773816dece3a9 [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:07.924417 5027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 075b57115a0143ae996773816dece3a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 075b57115a0143ae996773816dece3a9. Latest consensus state: current_term: 1 leader_uuid: "075b57115a0143ae996773816dece3a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "075b57115a0143ae996773816dece3a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39141 } } }
I20260928 22:53:07.924492 5027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 075b57115a0143ae996773816dece3a9 [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:07.924587 5036 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:53:07.925223 5036 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:53:07.927748 5036 catalog_manager.cc:1347] Generated new cluster ID: d97a539e09074a5eb88de3012459ee60
I20260928 22:53:07.927832 5036 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 22:53:07 dist-test-slave-56vw krb5kdc[4941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 22:53:07.936932 5035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:07.924051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51035 (local address 127.0.0.1:39141)
0928 22:53:07.924226 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:07.924238 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:07.924285 (+ 47us) server_negotiation.cc:407] Connection header received
0928 22:53:07.924439 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:07.924449 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:07.924535 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:07.925800 (+ 1265us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:07.925910 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:07.927325 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:07.928509 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:07.929211 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:07.929328 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:07.932723 (+ 3395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:07.932729 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:07.932730 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:07.932745 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:07.935072 (+ 2327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:07.935516 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:07.935520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:07.935521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:07.935668 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:07.935903 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:07.935903 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:07.935904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:07.936251 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:07.936408 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:07.936530 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 22:53:07.936678 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":76,"threads_started":1}
I20260928 22:53:07.939299 16870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:53:07.945216 5036 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:53:07.946000 5036 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:53:07.951615 5035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:07.940841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56279 (local address 127.0.0.1:39141)
0928 22:53:07.940881 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:07.940883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:07.943300 (+ 2417us) server_negotiation.cc:407] Connection header received
0928 22:53:07.943372 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:07.943374 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:07.943387 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:07.943458 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:07.944231 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:07.945037 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:07.946202 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:07.946564 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:07.946640 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:07.948279 (+ 1639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:07.948281 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:07.948282 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:07.948290 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:07.950112 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:07.950562 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:07.950563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:07.950563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:07.950628 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:07.950848 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:07.950849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:07.950849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:07.950997 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:07.951154 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:07.951408 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 22:53:07.951513 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":13,"server-negotiator.queue_time_us":19}
I20260928 22:53:07.951625 5036 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 075b57115a0143ae996773816dece3a9: Generated new TSK 0
I20260928 22:53:07.952186 5036 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:53:07.959445 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 4954
2026-09-28T22:53:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1358 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635733329745-16870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:53:08 dist-test-slave-56vw krb5kdc[5053](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:53:08 dist-test-slave-56vw krb5kdc[5053](info): set up 2 sockets
Sep 28 22:53:08 dist-test-slave-56vw krb5kdc[5053](info): commencing operation
krb5kdc: starting...
W20260928 22:53:09.111599 16870 mini_kdc.cc:121] Time spent starting KDC: real 1.133s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Sep 28 22:53:09 dist-test-slave-56vw krb5kdc[5053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:53:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:53:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635733329745-16870-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635733329745-16870-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 22:53:09.246557 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33335
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635733329745-16870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:09.332602 5066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:09.332859 5066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:09.332903 5066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:09.334075 5066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:53:09.334153 5066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:09.334177 5066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:09.334205 5066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:53:09.334235 5066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:53:09.335757 5066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635733329745-16870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:09.336073 5066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:09.336355 5066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:09.341710 5072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:09.341711 5071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:09.341750 5074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:09.342120 5066 server_base.cc:1034] running on GCE node
I20260928 22:53:09.342316 5066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:09.342608 5066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:09.343777 5066 hybrid_clock.cc:648] HybridClock initialized: now 1790635989343745 us; error 47 us; skew 500 ppm
Sep 28 22:53:09 dist-test-slave-56vw krb5kdc[5053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:09.346513 5066 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 22:53:09.347018 5066 webserver.cc:460] Webserver started at http://127.0.0.1:44091/ using document root <none> and password file <none>
I20260928 22:53:09.347304 5066 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:09.347353 5066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:09.347455 5066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:09.348508 5066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "33850f2a18994316972db4f2acd3e4fd"
format_stamp: "Formatted at 2026-09-28 22:53:09 on dist-test-slave-56vw"
server_key: "959836cd93cd07a0d85bdce3d2ca2fb8"
server_key_iv: "5fbd41ede1bfa75a2256d42f4708eecc"
server_key_version: "encryptionkey@0"
I20260928 22:53:09.348807 5066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "33850f2a18994316972db4f2acd3e4fd"
format_stamp: "Formatted at 2026-09-28 22:53:09 on dist-test-slave-56vw"
server_key: "959836cd93cd07a0d85bdce3d2ca2fb8"
server_key_iv: "5fbd41ede1bfa75a2256d42f4708eecc"
server_key_version: "encryptionkey@0"
I20260928 22:53:09.350114 5066 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:53:09.350967 5081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:09.351382 5066 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:53:09.351547 5066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "33850f2a18994316972db4f2acd3e4fd"
format_stamp: "Formatted at 2026-09-28 22:53:09 on dist-test-slave-56vw"
server_key: "959836cd93cd07a0d85bdce3d2ca2fb8"
server_key_iv: "5fbd41ede1bfa75a2256d42f4708eecc"
server_key_version: "encryptionkey@0"
I20260928 22:53:09.351629 5066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:09.360816 5066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:09.363180 5066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:09.363358 5066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:09.367764 5066 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33335
I20260928 22:53:09.367902 5133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33335 every 8 connection(s)
I20260928 22:53:09.368197 5066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:53:09.369035 5134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:09.371557 5134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33850f2a18994316972db4f2acd3e4fd: Bootstrap starting.
I20260928 22:53:09.371841 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 5066
I20260928 22:53:09.372002 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:53:09.372175 16870 external_mini_cluster.cc:1428] Setting key bfb21ce7b9e72d8af271f6c9f8e00592
I20260928 22:53:09.372330 5134 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33850f2a18994316972db4f2acd3e4fd: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:09.372720 5134 log.cc:824] T 00000000000000000000000000000000 P 33850f2a18994316972db4f2acd3e4fd: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:09.373517 5134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33850f2a18994316972db4f2acd3e4fd: No bootstrap required, opened a new log
I20260928 22:53:09.375705 5134 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33850f2a18994316972db4f2acd3e4fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33850f2a18994316972db4f2acd3e4fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33335 } }
I20260928 22:53:09.375886 5134 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33850f2a18994316972db4f2acd3e4fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:09.375918 5134 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33850f2a18994316972db4f2acd3e4fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33850f2a18994316972db4f2acd3e4fd, State: Initialized, Role: FOLLOWER
I20260928 22:53:09.376058 5134 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33850f2a18994316972db4f2acd3e4fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33850f2a18994316972db4f2acd3e4fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33335 } }
I20260928 22:53:09.376130 5134 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33850f2a18994316972db4f2acd3e4fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:53:09.376163 5134 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33850f2a18994316972db4f2acd3e4fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:53:09.376221 5134 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33850f2a18994316972db4f2acd3e4fd [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:09.376941 5134 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33850f2a18994316972db4f2acd3e4fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33850f2a18994316972db4f2acd3e4fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33335 } }
I20260928 22:53:09.377045 5134 leader_election.cc:302] T 00000000000000000000000000000000 P 33850f2a18994316972db4f2acd3e4fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33850f2a18994316972db4f2acd3e4fd; no voters:
I20260928 22:53:09.377213 5134 leader_election.cc:288] T 00000000000000000000000000000000 P 33850f2a18994316972db4f2acd3e4fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:53:09.377264 5139 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33850f2a18994316972db4f2acd3e4fd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:53:09.377405 5139 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33850f2a18994316972db4f2acd3e4fd [term 1 LEADER]: Becoming Leader. State: Replica: 33850f2a18994316972db4f2acd3e4fd, State: Running, Role: LEADER
I20260928 22:53:09.377544 5134 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33850f2a18994316972db4f2acd3e4fd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:53:09.377533 5139 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33850f2a18994316972db4f2acd3e4fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33850f2a18994316972db4f2acd3e4fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33335 } }
I20260928 22:53:09.378011 5141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33850f2a18994316972db4f2acd3e4fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33850f2a18994316972db4f2acd3e4fd. Latest consensus state: current_term: 1 leader_uuid: "33850f2a18994316972db4f2acd3e4fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33850f2a18994316972db4f2acd3e4fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33335 } } }
I20260928 22:53:09.378105 5141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33850f2a18994316972db4f2acd3e4fd [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:09.378716 5137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:09.372791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43805 (local address 127.0.0.1:33335)
0928 22:53:09.372952 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:09.372963 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:09.372999 (+ 36us) server_negotiation.cc:407] Connection header received
0928 22:53:09.373577 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:09.373587 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:09.373657 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:09.373778 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:09.374083 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:09.375017 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:09.376243 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:09.376845 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:09.376973 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:09.377724 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:09.377729 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:09.377729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:53:09.377962 (+ 233us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:09.378003 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:53:09.378018 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:09.378086 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:09.378178 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 22:53:09.378245 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":71,"threads_started":1}
I20260928 22:53:09.378612 5140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33850f2a18994316972db4f2acd3e4fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33850f2a18994316972db4f2acd3e4fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33850f2a18994316972db4f2acd3e4fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33335 } } }
I20260928 22:53:09.379420 5140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33850f2a18994316972db4f2acd3e4fd [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:09.379745 16870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260928 22:53:09.380019 5154 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 33850f2a18994316972db4f2acd3e4fd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:53:09.380072 5154 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:53:09.380118 5155 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:53:09.380286 5155 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:53:09.382019 5155 catalog_manager.cc:1347] Generated new cluster ID: 78362d2abe084ced8a02b35659e6161b
I20260928 22:53:09.382067 5155 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:53:09.384488 5137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:09.381381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54817 (local address 127.0.0.1:33335)
0928 22:53:09.381504 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:09.381506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:09.381517 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:53:09.381600 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:09.381601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:09.381611 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:09.381671 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:09.381865 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:09.382388 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:09.383224 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:09.383422 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:09.383514 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:09.384224 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:09.384226 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:09.384226 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:53:09.384230 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:09.384245 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:53:09.384249 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:09.384302 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:09.384357 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 22:53:09.384380 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260928 22:53:09.402503 5155 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:53:09.403249 5155 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:53:09.414181 5155 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33850f2a18994316972db4f2acd3e4fd: Generated new TSK 0
I20260928 22:53:09.414438 5155 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:53:09.421633 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 5066
2026-09-28T22:53:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1462 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635733329745-16870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:53:09 dist-test-slave-56vw krb5kdc[5165](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:53:09 dist-test-slave-56vw krb5kdc[5165](info): set up 2 sockets
Sep 28 22:53:09 dist-test-slave-56vw krb5kdc[5165](info): commencing operation
krb5kdc: starting...
W20260928 22:53:10.484725 16870 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Sep 28 22:53:10 dist-test-slave-56vw krb5kdc[5165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:53:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:53:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635733329745-16870-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635733329745-16870-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 22:53:10.612361 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44005
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635733329745-16870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:10.696612 5178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:10.696838 5178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:10.696867 5178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:10.698148 5178 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:53:10.698195 5178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:10.698210 5178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:10.698223 5178 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:53:10.698235 5178 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:53:10.699949 5178 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635733329745-16870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:10.700227 5178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:10.700479 5178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:10.705595 5184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:10.705560 5183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:10.705579 5186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:10.705891 5178 server_base.cc:1034] running on GCE node
I20260928 22:53:10.706118 5178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:10.706389 5178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:10.707572 5178 hybrid_clock.cc:648] HybridClock initialized: now 1790635990707557 us; error 27 us; skew 500 ppm
Sep 28 22:53:10 dist-test-slave-56vw krb5kdc[5165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:10.723027 5178 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 22:53:10.723671 5178 webserver.cc:460] Webserver started at http://127.0.0.1:35453/ using document root <none> and password file <none>
I20260928 22:53:10.723958 5178 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:10.724048 5178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:10.724202 5178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:10.725327 5178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "547a9b218b724cffae8b39bd8b307a74"
format_stamp: "Formatted at 2026-09-28 22:53:10 on dist-test-slave-56vw"
server_key: "29ae3c643b69d6011856f03be7d06256"
server_key_iv: "7dec731b6d77ae9976c27076f38f523d"
server_key_version: "encryptionkey@0"
I20260928 22:53:10.725708 5178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "547a9b218b724cffae8b39bd8b307a74"
format_stamp: "Formatted at 2026-09-28 22:53:10 on dist-test-slave-56vw"
server_key: "29ae3c643b69d6011856f03be7d06256"
server_key_iv: "7dec731b6d77ae9976c27076f38f523d"
server_key_version: "encryptionkey@0"
I20260928 22:53:10.727195 5178 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 22:53:10.728008 5193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:10.728199 5178 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:53:10.728305 5178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "547a9b218b724cffae8b39bd8b307a74"
format_stamp: "Formatted at 2026-09-28 22:53:10 on dist-test-slave-56vw"
server_key: "29ae3c643b69d6011856f03be7d06256"
server_key_iv: "7dec731b6d77ae9976c27076f38f523d"
server_key_version: "encryptionkey@0"
I20260928 22:53:10.728389 5178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:10.731937 5178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:10.733899 5178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:10.734092 5178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:10.738245 5178 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44005
I20260928 22:53:10.738327 5245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44005 every 8 connection(s)
I20260928 22:53:10.738660 5178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:53:10.739825 5246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:10.742244 5246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 547a9b218b724cffae8b39bd8b307a74: Bootstrap starting.
I20260928 22:53:10.742837 5246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 547a9b218b724cffae8b39bd8b307a74: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:10.743175 5246 log.cc:824] T 00000000000000000000000000000000 P 547a9b218b724cffae8b39bd8b307a74: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:10.743923 5246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 547a9b218b724cffae8b39bd8b307a74: No bootstrap required, opened a new log
I20260928 22:53:10.745527 5246 raft_consensus.cc:348] T 00000000000000000000000000000000 P 547a9b218b724cffae8b39bd8b307a74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547a9b218b724cffae8b39bd8b307a74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44005 } }
I20260928 22:53:10.745668 5246 raft_consensus.cc:374] T 00000000000000000000000000000000 P 547a9b218b724cffae8b39bd8b307a74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:10.745690 5246 raft_consensus.cc:729] T 00000000000000000000000000000000 P 547a9b218b724cffae8b39bd8b307a74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 547a9b218b724cffae8b39bd8b307a74, State: Initialized, Role: FOLLOWER
I20260928 22:53:10.745862 5246 consensus_queue.cc:260] T 00000000000000000000000000000000 P 547a9b218b724cffae8b39bd8b307a74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547a9b218b724cffae8b39bd8b307a74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44005 } }
I20260928 22:53:10.745923 5246 raft_consensus.cc:388] T 00000000000000000000000000000000 P 547a9b218b724cffae8b39bd8b307a74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:53:10.745975 5246 raft_consensus.cc:482] T 00000000000000000000000000000000 P 547a9b218b724cffae8b39bd8b307a74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:53:10.746030 5246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 547a9b218b724cffae8b39bd8b307a74 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:10.746595 5246 raft_consensus.cc:504] T 00000000000000000000000000000000 P 547a9b218b724cffae8b39bd8b307a74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547a9b218b724cffae8b39bd8b307a74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44005 } }
I20260928 22:53:10.746707 5246 leader_election.cc:302] T 00000000000000000000000000000000 P 547a9b218b724cffae8b39bd8b307a74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 547a9b218b724cffae8b39bd8b307a74; no voters:
I20260928 22:53:10.746904 5246 leader_election.cc:288] T 00000000000000000000000000000000 P 547a9b218b724cffae8b39bd8b307a74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:53:10.746953 5249 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 547a9b218b724cffae8b39bd8b307a74 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:53:10.747188 5249 raft_consensus.cc:686] T 00000000000000000000000000000000 P 547a9b218b724cffae8b39bd8b307a74 [term 1 LEADER]: Becoming Leader. State: Replica: 547a9b218b724cffae8b39bd8b307a74, State: Running, Role: LEADER
I20260928 22:53:10.747196 5246 sys_catalog.cc:564] T 00000000000000000000000000000000 P 547a9b218b724cffae8b39bd8b307a74 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:53:10.747349 5249 consensus_queue.cc:237] T 00000000000000000000000000000000 P 547a9b218b724cffae8b39bd8b307a74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547a9b218b724cffae8b39bd8b307a74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44005 } }
I20260928 22:53:10.747714 5250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 547a9b218b724cffae8b39bd8b307a74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "547a9b218b724cffae8b39bd8b307a74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547a9b218b724cffae8b39bd8b307a74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44005 } } }
I20260928 22:53:10.747808 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 5178
I20260928 22:53:10.747725 5251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 547a9b218b724cffae8b39bd8b307a74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 547a9b218b724cffae8b39bd8b307a74. Latest consensus state: current_term: 1 leader_uuid: "547a9b218b724cffae8b39bd8b307a74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547a9b218b724cffae8b39bd8b307a74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44005 } } }
I20260928 22:53:10.747871 5250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 547a9b218b724cffae8b39bd8b307a74 [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:10.747874 5251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 547a9b218b724cffae8b39bd8b307a74 [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:10.747958 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:53:10.748096 5253 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:53:10.748167 16870 external_mini_cluster.cc:1428] Setting key 0384164e1143fc2b327cda11cdfa487c
I20260928 22:53:10.748849 5253 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:53:10.751019 5266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:10.748830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48597 (local address 127.0.0.1:44005)
0928 22:53:10.749510 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:10.749517 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:10.749541 (+ 24us) server_negotiation.cc:407] Connection header received
0928 22:53:10.749629 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:10.749634 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:10.749658 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:10.749800 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:10.750543 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:10.750547 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:10.750547 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:53:10.750552 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:10.750600 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:53:10.750615 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:10.750693 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:10.750783 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 22:53:10.750835 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":66,"threads_started":1}
I20260928 22:53:10.751214 5253 catalog_manager.cc:1347] Generated new cluster ID: e52b0e39a9b94fd19fcf727981acf202
I20260928 22:53:10.751288 5253 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:53:10.752125 16870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:53:10.754669 5266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:10.753538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54379 (local address 127.0.0.1:44005)
0928 22:53:10.753577 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:10.753579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:10.753738 (+ 159us) server_negotiation.cc:407] Connection header received
0928 22:53:10.753767 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:10.753768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:10.753774 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:10.753853 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:10.754450 (+ 597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:10.754452 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:10.754452 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:53:10.754456 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:10.754473 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:53:10.754478 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:10.754539 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:10.754594 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 22:53:10.754606 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 22:53:10.772374 5253 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:53:10.772586 5253 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:53:10.776142 5253 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 547a9b218b724cffae8b39bd8b307a74: Generated new TSK 0
I20260928 22:53:10.776358 5253 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:53:10.784102 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 5178
2026-09-28T22:53:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1360 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635733329745-16870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:53:10 dist-test-slave-56vw krb5kdc[5277](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:53:10 dist-test-slave-56vw krb5kdc[5277](info): set up 2 sockets
Sep 28 22:53:10 dist-test-slave-56vw krb5kdc[5277](info): commencing operation
krb5kdc: starting...
W20260928 22:53:11.855362 16870 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Sep 28 22:53:11 dist-test-slave-56vw krb5kdc[5277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:53:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:53:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635733329745-16870-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635733329745-16870-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 22:53:11.983224 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37183
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635733329745-16870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:12.070356 5290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:12.070595 5290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:12.070623 5290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:12.071998 5290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:53:12.072052 5290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:12.072067 5290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:12.072079 5290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:53:12.072091 5290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:53:12.073674 5290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635733329745-16870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:12.073935 5290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:12.074193 5290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:12.079905 5295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:12.079900 5298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:12.079991 5296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:12.080320 5290 server_base.cc:1034] running on GCE node
I20260928 22:53:12.080600 5290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:12.080863 5290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:12.082015 5290 hybrid_clock.cc:648] HybridClock initialized: now 1790635992082001 us; error 31 us; skew 500 ppm
Sep 28 22:53:12 dist-test-slave-56vw krb5kdc[5277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:12.084724 5290 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 22:53:12.085248 5290 webserver.cc:460] Webserver started at http://127.0.0.1:41179/ using document root <none> and password file <none>
I20260928 22:53:12.085495 5290 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:12.085567 5290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:12.085718 5290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:12.086848 5290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "1f5ba42b04d04825b8de62d796928a93"
format_stamp: "Formatted at 2026-09-28 22:53:12 on dist-test-slave-56vw"
server_key: "376e56149428aa3b8c87545853ddb8c3"
server_key_iv: "1f7f18b7c8a423e4296716bf77d426d1"
server_key_version: "encryptionkey@0"
I20260928 22:53:12.087253 5290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "1f5ba42b04d04825b8de62d796928a93"
format_stamp: "Formatted at 2026-09-28 22:53:12 on dist-test-slave-56vw"
server_key: "376e56149428aa3b8c87545853ddb8c3"
server_key_iv: "1f7f18b7c8a423e4296716bf77d426d1"
server_key_version: "encryptionkey@0"
I20260928 22:53:12.088809 5290 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:53:12.089625 5305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:12.089819 5290 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:53:12.089931 5290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "1f5ba42b04d04825b8de62d796928a93"
format_stamp: "Formatted at 2026-09-28 22:53:12 on dist-test-slave-56vw"
server_key: "376e56149428aa3b8c87545853ddb8c3"
server_key_iv: "1f7f18b7c8a423e4296716bf77d426d1"
server_key_version: "encryptionkey@0"
I20260928 22:53:12.090013 5290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:12.113215 5290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:12.115703 5290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:12.115885 5290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:12.120188 5290 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37183
I20260928 22:53:12.120221 5357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37183 every 8 connection(s)
I20260928 22:53:12.120604 5290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:53:12.121544 5358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:12.124125 5358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f5ba42b04d04825b8de62d796928a93: Bootstrap starting.
I20260928 22:53:12.124688 5358 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f5ba42b04d04825b8de62d796928a93: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:12.124931 5358 log.cc:824] T 00000000000000000000000000000000 P 1f5ba42b04d04825b8de62d796928a93: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:12.125674 5358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f5ba42b04d04825b8de62d796928a93: No bootstrap required, opened a new log
I20260928 22:53:12.127349 5358 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f5ba42b04d04825b8de62d796928a93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f5ba42b04d04825b8de62d796928a93" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37183 } }
I20260928 22:53:12.127501 5358 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f5ba42b04d04825b8de62d796928a93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:12.127524 5358 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f5ba42b04d04825b8de62d796928a93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f5ba42b04d04825b8de62d796928a93, State: Initialized, Role: FOLLOWER
I20260928 22:53:12.127628 5358 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f5ba42b04d04825b8de62d796928a93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f5ba42b04d04825b8de62d796928a93" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37183 } }
I20260928 22:53:12.127681 5358 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f5ba42b04d04825b8de62d796928a93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:53:12.127702 5358 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f5ba42b04d04825b8de62d796928a93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:53:12.127737 5358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f5ba42b04d04825b8de62d796928a93 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:12.128235 5358 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f5ba42b04d04825b8de62d796928a93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f5ba42b04d04825b8de62d796928a93" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37183 } }
I20260928 22:53:12.128317 5358 leader_election.cc:302] T 00000000000000000000000000000000 P 1f5ba42b04d04825b8de62d796928a93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f5ba42b04d04825b8de62d796928a93; no voters:
I20260928 22:53:12.128453 5358 leader_election.cc:288] T 00000000000000000000000000000000 P 1f5ba42b04d04825b8de62d796928a93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:53:12.128520 5361 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f5ba42b04d04825b8de62d796928a93 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:53:12.128717 5361 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f5ba42b04d04825b8de62d796928a93 [term 1 LEADER]: Becoming Leader. State: Replica: 1f5ba42b04d04825b8de62d796928a93, State: Running, Role: LEADER
I20260928 22:53:12.128754 5358 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f5ba42b04d04825b8de62d796928a93 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:53:12.128870 5361 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f5ba42b04d04825b8de62d796928a93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f5ba42b04d04825b8de62d796928a93" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37183 } }
I20260928 22:53:12.128974 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 5290
I20260928 22:53:12.129125 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:53:12.129351 16870 external_mini_cluster.cc:1428] Setting key 1d447c3ebe028011a6ad7e7279f792e9
I20260928 22:53:12.129642 5363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f5ba42b04d04825b8de62d796928a93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f5ba42b04d04825b8de62d796928a93. Latest consensus state: current_term: 1 leader_uuid: "1f5ba42b04d04825b8de62d796928a93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f5ba42b04d04825b8de62d796928a93" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37183 } } }
I20260928 22:53:12.129751 5363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f5ba42b04d04825b8de62d796928a93 [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:12.130124 5371 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:53:12.130411 5362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f5ba42b04d04825b8de62d796928a93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f5ba42b04d04825b8de62d796928a93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f5ba42b04d04825b8de62d796928a93" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37183 } } }
I20260928 22:53:12.130667 5362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f5ba42b04d04825b8de62d796928a93 [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:12.130813 5371 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:53:12.132522 5371 catalog_manager.cc:1347] Generated new cluster ID: 843b01c9061a4af88b0930d25221584f
I20260928 22:53:12.132586 5371 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 22:53:12 dist-test-slave-56vw krb5kdc[5277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 22:53:12.140419 5377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:12.130086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.132:33411 (local address 127.0.0.1:37183)
0928 22:53:12.130263 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:12.130273 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:12.130803 (+ 530us) server_negotiation.cc:407] Connection header received
0928 22:53:12.130978 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:12.130984 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:12.131033 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:12.131232 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:12.131948 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:12.133141 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:12.133927 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:12.134371 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:12.134491 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:12.136910 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:12.136915 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:12.136915 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:12.136936 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:12.138695 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:12.139160 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:12.139168 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:12.139168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:12.139270 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:12.139512 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:12.139513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:12.139513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:12.139786 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:12.139963 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:12.140092 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 22:53:12.140201 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":76,"threads_started":1}
I20260928 22:53:12.141417 16870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:53:12.149842 5377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:12.142795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.132:43659 (local address 127.0.0.1:37183)
0928 22:53:12.142837 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:12.142839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:12.142993 (+ 154us) server_negotiation.cc:407] Connection header received
0928 22:53:12.143095 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:12.143096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:12.143107 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:12.143155 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:12.143699 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:12.144233 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:12.144974 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:12.145207 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:12.146397 (+ 1190us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:12.146879 (+ 482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:12.146880 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:12.146881 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:12.146888 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:12.148548 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:12.148967 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:12.148968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:12.148969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:12.149035 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:12.149302 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:12.149303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:12.149303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:12.149430 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:12.149550 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:12.149677 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 22:53:12.149739 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:53:12.165959 5371 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:53:12.166569 5371 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:53:12.176239 5371 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f5ba42b04d04825b8de62d796928a93: Generated new TSK 0
I20260928 22:53:12.176576 5371 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:53:12.187765 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 5290
W20260928 22:53:12.192550 5275 connection.cc:535] client connection to 127.0.0.1:37183 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37183): Connection reset by peer (error 104)
2026-09-28T22:53:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1404 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635733329745-16870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:53:12 dist-test-slave-56vw krb5kdc[5389](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:53:12 dist-test-slave-56vw krb5kdc[5389](info): set up 2 sockets
Sep 28 22:53:12 dist-test-slave-56vw krb5kdc[5389](info): commencing operation
krb5kdc: starting...
W20260928 22:53:13.227337 16870 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Sep 28 22:53:13 dist-test-slave-56vw krb5kdc[5389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:53:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:53:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635733329745-16870-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635733329745-16870-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 22:53:13.352255 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33327
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635733329745-16870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:13.434293 5402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:13.434567 5402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:13.434619 5402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:13.435950 5402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:53:13.436040 5402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:13.436064 5402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:13.436086 5402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:53:13.436117 5402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:53:13.437682 5402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635733329745-16870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:13.438001 5402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:13.438243 5402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:13.443380 5407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:13.443454 5402 server_base.cc:1034] running on GCE node
W20260928 22:53:13.443391 5408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:13.443423 5410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:13.443804 5402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:13.444104 5402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:13.445269 5402 hybrid_clock.cc:648] HybridClock initialized: now 1790635993445265 us; error 38 us; skew 500 ppm
Sep 28 22:53:13 dist-test-slave-56vw krb5kdc[5389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:13.447893 5402 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 22:53:13.448329 5402 webserver.cc:460] Webserver started at http://127.0.0.1:41057/ using document root <none> and password file <none>
I20260928 22:53:13.448556 5402 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:13.448624 5402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:13.448773 5402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:13.449927 5402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "11419ac9853a4ea2baa5e6d1097f2919"
format_stamp: "Formatted at 2026-09-28 22:53:13 on dist-test-slave-56vw"
server_key: "b59f0527a663ae4e5b87e2da8c6a7f48"
server_key_iv: "02470ebbd818fa9233f4c84dd34fba6e"
server_key_version: "encryptionkey@0"
I20260928 22:53:13.450310 5402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "11419ac9853a4ea2baa5e6d1097f2919"
format_stamp: "Formatted at 2026-09-28 22:53:13 on dist-test-slave-56vw"
server_key: "b59f0527a663ae4e5b87e2da8c6a7f48"
server_key_iv: "02470ebbd818fa9233f4c84dd34fba6e"
server_key_version: "encryptionkey@0"
I20260928 22:53:13.451771 5402 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 22:53:13.452742 5417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:13.452960 5402 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:53:13.453076 5402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "11419ac9853a4ea2baa5e6d1097f2919"
format_stamp: "Formatted at 2026-09-28 22:53:13 on dist-test-slave-56vw"
server_key: "b59f0527a663ae4e5b87e2da8c6a7f48"
server_key_iv: "02470ebbd818fa9233f4c84dd34fba6e"
server_key_version: "encryptionkey@0"
I20260928 22:53:13.453152 5402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:13.476761 5402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:13.478890 5402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:13.479118 5402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:13.483198 5402 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33327
I20260928 22:53:13.483278 5469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33327 every 8 connection(s)
I20260928 22:53:13.483655 5402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:53:13.484601 5470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:13.486900 5470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11419ac9853a4ea2baa5e6d1097f2919: Bootstrap starting.
I20260928 22:53:13.487514 5470 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11419ac9853a4ea2baa5e6d1097f2919: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:13.487545 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 5402
I20260928 22:53:13.487672 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:53:13.487898 16870 external_mini_cluster.cc:1428] Setting key 9fb52f0d8c49846471adc8f0a6405562
I20260928 22:53:13.487882 5470 log.cc:824] T 00000000000000000000000000000000 P 11419ac9853a4ea2baa5e6d1097f2919: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:13.488751 5470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11419ac9853a4ea2baa5e6d1097f2919: No bootstrap required, opened a new log
I20260928 22:53:13.491252 5470 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11419ac9853a4ea2baa5e6d1097f2919 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11419ac9853a4ea2baa5e6d1097f2919" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33327 } }
I20260928 22:53:13.491431 5470 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11419ac9853a4ea2baa5e6d1097f2919 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:13.491470 5470 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11419ac9853a4ea2baa5e6d1097f2919 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11419ac9853a4ea2baa5e6d1097f2919, State: Initialized, Role: FOLLOWER
I20260928 22:53:13.491597 5470 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11419ac9853a4ea2baa5e6d1097f2919 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11419ac9853a4ea2baa5e6d1097f2919" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33327 } }
I20260928 22:53:13.491683 5470 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11419ac9853a4ea2baa5e6d1097f2919 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:53:13.491720 5470 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11419ac9853a4ea2baa5e6d1097f2919 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:53:13.491783 5470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11419ac9853a4ea2baa5e6d1097f2919 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:13.492410 5470 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11419ac9853a4ea2baa5e6d1097f2919 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11419ac9853a4ea2baa5e6d1097f2919" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33327 } }
I20260928 22:53:13.492542 5470 leader_election.cc:302] T 00000000000000000000000000000000 P 11419ac9853a4ea2baa5e6d1097f2919 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11419ac9853a4ea2baa5e6d1097f2919; no voters:
I20260928 22:53:13.492718 5470 leader_election.cc:288] T 00000000000000000000000000000000 P 11419ac9853a4ea2baa5e6d1097f2919 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:53:13.492749 5475 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11419ac9853a4ea2baa5e6d1097f2919 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:53:13.492954 5475 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11419ac9853a4ea2baa5e6d1097f2919 [term 1 LEADER]: Becoming Leader. State: Replica: 11419ac9853a4ea2baa5e6d1097f2919, State: Running, Role: LEADER
I20260928 22:53:13.492987 5470 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11419ac9853a4ea2baa5e6d1097f2919 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:53:13.493046 5475 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11419ac9853a4ea2baa5e6d1097f2919 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11419ac9853a4ea2baa5e6d1097f2919" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33327 } }
I20260928 22:53:13.493393 5475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11419ac9853a4ea2baa5e6d1097f2919 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11419ac9853a4ea2baa5e6d1097f2919. Latest consensus state: current_term: 1 leader_uuid: "11419ac9853a4ea2baa5e6d1097f2919" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11419ac9853a4ea2baa5e6d1097f2919" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33327 } } }
I20260928 22:53:13.493526 5475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11419ac9853a4ea2baa5e6d1097f2919 [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:13.493799 5482 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:53:13.494139 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:13.488502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.132:40465 (local address 127.0.0.1:33327)
0928 22:53:13.488671 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:13.488679 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:13.488705 (+ 26us) server_negotiation.cc:407] Connection header received
0928 22:53:13.489213 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:13.489221 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:13.489277 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:13.489395 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:13.489635 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:13.490510 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:13.491796 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:13.492267 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:13.492383 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:13.493287 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:13.493292 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:13.493292 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:53:13.493505 (+ 213us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:13.493545 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:53:13.493563 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:13.493652 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:13.493921 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 22:53:13.493951 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":81,"threads_started":1}
I20260928 22:53:13.494534 5476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11419ac9853a4ea2baa5e6d1097f2919 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11419ac9853a4ea2baa5e6d1097f2919" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11419ac9853a4ea2baa5e6d1097f2919" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33327 } } }
I20260928 22:53:13.494632 5476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11419ac9853a4ea2baa5e6d1097f2919 [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:13.494720 5482 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:53:13.495186 16870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:53:13.496519 5482 catalog_manager.cc:1347] Generated new cluster ID: 2f98471488db47c48650d6597ad5f571
I20260928 22:53:13.496692 5482 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:53:13.503271 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:13.499436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.132:52137 (local address 127.0.0.1:33327)
0928 22:53:13.499474 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:13.499476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:13.499693 (+ 217us) server_negotiation.cc:407] Connection header received
0928 22:53:13.499729 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:13.499730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:13.499743 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:13.499808 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:13.499948 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:13.500713 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:13.501728 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:13.502083 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:13.502160 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:13.502944 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:13.502945 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:13.502946 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:53:13.502951 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:13.502978 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:53:13.502983 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:13.503088 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:13.503151 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 22:53:13.503163 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:53:13.507916 5482 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:53:13.508637 5482 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:53:13.521589 5482 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11419ac9853a4ea2baa5e6d1097f2919: Generated new TSK 0
I20260928 22:53:13.521835 5482 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:53:13.524020 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 5402
W20260928 22:53:13.528286 5387 connection.cc:535] client connection to 127.0.0.1:33327 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33327): Connection reset by peer (error 104)
2026-09-28T22:53:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1335 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635733329745-16870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:53:13 dist-test-slave-56vw krb5kdc[5501](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:53:13 dist-test-slave-56vw krb5kdc[5501](info): set up 2 sockets
Sep 28 22:53:13 dist-test-slave-56vw krb5kdc[5501](info): commencing operation
krb5kdc: starting...
W20260928 22:53:14.571775 16870 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Sep 28 22:53:14 dist-test-slave-56vw krb5kdc[5501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:53:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:53:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635733329745-16870-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635733329745-16870-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 22:53:14.699494 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39431
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635733329745-16870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:14.783354 5514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:14.783630 5514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:14.783674 5514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:14.784960 5514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:53:14.785044 5514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:14.785068 5514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:14.785089 5514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:53:14.785121 5514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:53:14.786746 5514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635733329745-16870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:14.787108 5514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:14.787374 5514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:14.792634 5520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:14.792637 5522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:14.792779 5519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:14.792913 5514 server_base.cc:1034] running on GCE node
I20260928 22:53:14.793169 5514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:14.793515 5514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:14.794723 5514 hybrid_clock.cc:648] HybridClock initialized: now 1790635994794700 us; error 44 us; skew 500 ppm
Sep 28 22:53:14 dist-test-slave-56vw krb5kdc[5501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:14.798743 5514 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 22:53:14.799350 5514 webserver.cc:460] Webserver started at http://127.0.0.1:41171/ using document root <none> and password file <none>
I20260928 22:53:14.799680 5514 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:14.799759 5514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:14.799925 5514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:14.801024 5514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "c4a025ce860b4660851310a1fec0e7d4"
format_stamp: "Formatted at 2026-09-28 22:53:14 on dist-test-slave-56vw"
server_key: "0cab19d4abb4c1b7ae6040d59ea71602"
server_key_iv: "88c930d0a23562ca75870b3933de0490"
server_key_version: "encryptionkey@0"
I20260928 22:53:14.801487 5514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "c4a025ce860b4660851310a1fec0e7d4"
format_stamp: "Formatted at 2026-09-28 22:53:14 on dist-test-slave-56vw"
server_key: "0cab19d4abb4c1b7ae6040d59ea71602"
server_key_iv: "88c930d0a23562ca75870b3933de0490"
server_key_version: "encryptionkey@0"
I20260928 22:53:14.802908 5514 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:53:14.803809 5529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:14.804013 5514 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:53:14.804116 5514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "c4a025ce860b4660851310a1fec0e7d4"
format_stamp: "Formatted at 2026-09-28 22:53:14 on dist-test-slave-56vw"
server_key: "0cab19d4abb4c1b7ae6040d59ea71602"
server_key_iv: "88c930d0a23562ca75870b3933de0490"
server_key_version: "encryptionkey@0"
I20260928 22:53:14.804203 5514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:14.807699 5514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:14.815716 5514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:14.815893 5514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:14.819963 5514 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39431
I20260928 22:53:14.819998 5581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39431 every 8 connection(s)
I20260928 22:53:14.820353 5514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:53:14.821244 5582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:14.823671 5582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4a025ce860b4660851310a1fec0e7d4: Bootstrap starting.
I20260928 22:53:14.824256 5582 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4a025ce860b4660851310a1fec0e7d4: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:14.824555 5582 log.cc:824] T 00000000000000000000000000000000 P c4a025ce860b4660851310a1fec0e7d4: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:14.824891 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 5514
I20260928 22:53:14.824991 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:53:14.825193 16870 external_mini_cluster.cc:1428] Setting key 268133fe819eeb9d844a6affb48d3c28
I20260928 22:53:14.825431 5582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4a025ce860b4660851310a1fec0e7d4: No bootstrap required, opened a new log
I20260928 22:53:14.827840 5582 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4a025ce860b4660851310a1fec0e7d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4a025ce860b4660851310a1fec0e7d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39431 } }
I20260928 22:53:14.828061 5582 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4a025ce860b4660851310a1fec0e7d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:14.828131 5582 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4a025ce860b4660851310a1fec0e7d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4a025ce860b4660851310a1fec0e7d4, State: Initialized, Role: FOLLOWER
I20260928 22:53:14.828280 5582 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4a025ce860b4660851310a1fec0e7d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4a025ce860b4660851310a1fec0e7d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39431 } }
I20260928 22:53:14.828399 5582 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4a025ce860b4660851310a1fec0e7d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:53:14.828442 5582 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4a025ce860b4660851310a1fec0e7d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:53:14.828470 5586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:14.825746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.132:55421 (local address 127.0.0.1:39431)
0928 22:53:14.825963 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:14.825974 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:14.826010 (+ 36us) server_negotiation.cc:407] Connection header received
0928 22:53:14.826872 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:14.826882 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:14.826918 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:14.827106 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:14.827781 (+ 675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:14.827785 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:14.827785 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:53:14.827790 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:14.827839 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:53:14.827853 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:14.827930 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:14.827988 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 22:53:14.828031 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":95,"threads_started":1}
I20260928 22:53:14.828496 5582 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4a025ce860b4660851310a1fec0e7d4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:14.829161 5582 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4a025ce860b4660851310a1fec0e7d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4a025ce860b4660851310a1fec0e7d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39431 } }
I20260928 22:53:14.829296 5582 leader_election.cc:302] T 00000000000000000000000000000000 P c4a025ce860b4660851310a1fec0e7d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4a025ce860b4660851310a1fec0e7d4; no voters:
I20260928 22:53:14.829468 16870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:53:14.829501 5582 leader_election.cc:288] T 00000000000000000000000000000000 P c4a025ce860b4660851310a1fec0e7d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:53:14.829524 5587 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4a025ce860b4660851310a1fec0e7d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:53:14.829627 5587 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4a025ce860b4660851310a1fec0e7d4 [term 1 LEADER]: Becoming Leader. State: Replica: c4a025ce860b4660851310a1fec0e7d4, State: Running, Role: LEADER
I20260928 22:53:14.829753 5582 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4a025ce860b4660851310a1fec0e7d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:53:14.829759 5587 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4a025ce860b4660851310a1fec0e7d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4a025ce860b4660851310a1fec0e7d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39431 } }
I20260928 22:53:14.830191 5588 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4a025ce860b4660851310a1fec0e7d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4a025ce860b4660851310a1fec0e7d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4a025ce860b4660851310a1fec0e7d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39431 } } }
I20260928 22:53:14.830397 5588 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4a025ce860b4660851310a1fec0e7d4 [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:14.830327 5589 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4a025ce860b4660851310a1fec0e7d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4a025ce860b4660851310a1fec0e7d4. Latest consensus state: current_term: 1 leader_uuid: "c4a025ce860b4660851310a1fec0e7d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4a025ce860b4660851310a1fec0e7d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39431 } } }
I20260928 22:53:14.830672 5589 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4a025ce860b4660851310a1fec0e7d4 [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:14.831094 5596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:53:14.831763 5596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:53:14.832564 5586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:14.831142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.132:39191 (local address 127.0.0.1:39431)
0928 22:53:14.831226 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:14.831228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:14.831359 (+ 131us) server_negotiation.cc:407] Connection header received
0928 22:53:14.831518 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:14.831519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:14.831525 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:14.831573 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:14.832328 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:14.832330 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:14.832330 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:53:14.832335 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:14.832352 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:53:14.832357 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:14.832417 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:14.832490 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 22:53:14.832506 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 22:53:14.833534 5596 catalog_manager.cc:1347] Generated new cluster ID: c8e2999381e14bbf9969e9a723561be6
I20260928 22:53:14.833583 5596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:53:14.845386 5596 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:53:14.845603 5596 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:53:14.848097 5596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4a025ce860b4660851310a1fec0e7d4: Generated new TSK 0
I20260928 22:53:14.848317 5596 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:53:14.855140 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 5514
2026-09-28T22:53:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1331 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635733329745-16870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:53:14 dist-test-slave-56vw krb5kdc[5613](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:53:14 dist-test-slave-56vw krb5kdc[5613](info): set up 2 sockets
Sep 28 22:53:14 dist-test-slave-56vw krb5kdc[5613](info): commencing operation
krb5kdc: starting...
W20260928 22:53:15.931397 16870 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Sep 28 22:53:16 dist-test-slave-56vw krb5kdc[5613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:53:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:53:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635733329745-16870-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635733329745-16870-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 22:53:16.074486 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45629
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635733329745-16870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:16.158514 5626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:16.158783 5626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:16.158824 5626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:16.160089 5626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:53:16.160161 5626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:16.160185 5626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:16.160214 5626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:53:16.160275 5626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:53:16.161810 5626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45629
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635733329745-16870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:16.162106 5626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:16.162355 5626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:16.167512 5632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:16.167512 5631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:16.167575 5634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:16.167541 5626 server_base.cc:1034] running on GCE node
I20260928 22:53:16.167937 5626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:16.168200 5626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:16.169416 5626 hybrid_clock.cc:648] HybridClock initialized: now 1790635996169400 us; error 31 us; skew 500 ppm
Sep 28 22:53:16 dist-test-slave-56vw krb5kdc[5613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:16.180794 5626 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 22:53:16.181310 5626 webserver.cc:460] Webserver started at http://127.0.0.1:42263/ using document root <none> and password file <none>
I20260928 22:53:16.181530 5626 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:16.181573 5626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:16.181671 5626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:16.182765 5626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "9ac5a019d552488d82dd806b6d9831fa"
format_stamp: "Formatted at 2026-09-28 22:53:16 on dist-test-slave-56vw"
server_key: "37e91ef2134f835f230d725d28fd6470"
server_key_iv: "8bf8beb261ec73f9ae9249b505513970"
server_key_version: "encryptionkey@0"
I20260928 22:53:16.183149 5626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "9ac5a019d552488d82dd806b6d9831fa"
format_stamp: "Formatted at 2026-09-28 22:53:16 on dist-test-slave-56vw"
server_key: "37e91ef2134f835f230d725d28fd6470"
server_key_iv: "8bf8beb261ec73f9ae9249b505513970"
server_key_version: "encryptionkey@0"
I20260928 22:53:16.184523 5626 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:53:16.185289 5641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:16.185467 5626 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 22:53:16.185554 5626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "9ac5a019d552488d82dd806b6d9831fa"
format_stamp: "Formatted at 2026-09-28 22:53:16 on dist-test-slave-56vw"
server_key: "37e91ef2134f835f230d725d28fd6470"
server_key_iv: "8bf8beb261ec73f9ae9249b505513970"
server_key_version: "encryptionkey@0"
I20260928 22:53:16.185608 5626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:16.198655 5626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:16.200764 5626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:16.200881 5626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:16.204937 5626 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45629
I20260928 22:53:16.205030 5693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45629 every 8 connection(s)
I20260928 22:53:16.205379 5626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:53:16.206344 5694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:16.208874 5694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ac5a019d552488d82dd806b6d9831fa: Bootstrap starting.
I20260928 22:53:16.209491 5694 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ac5a019d552488d82dd806b6d9831fa: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:16.209789 5694 log.cc:824] T 00000000000000000000000000000000 P 9ac5a019d552488d82dd806b6d9831fa: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:16.210096 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 5626
I20260928 22:53:16.210232 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:53:16.210513 16870 external_mini_cluster.cc:1428] Setting key 1dc334d83965a9750927587702d74e5a
I20260928 22:53:16.210685 5694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ac5a019d552488d82dd806b6d9831fa: No bootstrap required, opened a new log
I20260928 22:53:16.212607 5694 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ac5a019d552488d82dd806b6d9831fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ac5a019d552488d82dd806b6d9831fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45629 } }
I20260928 22:53:16.212749 5694 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ac5a019d552488d82dd806b6d9831fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:16.212772 5694 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ac5a019d552488d82dd806b6d9831fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ac5a019d552488d82dd806b6d9831fa, State: Initialized, Role: FOLLOWER
I20260928 22:53:16.212913 5694 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ac5a019d552488d82dd806b6d9831fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ac5a019d552488d82dd806b6d9831fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45629 } }
I20260928 22:53:16.213027 5694 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ac5a019d552488d82dd806b6d9831fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:53:16.213070 5694 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ac5a019d552488d82dd806b6d9831fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:53:16.213145 5694 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ac5a019d552488d82dd806b6d9831fa [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:16.214092 5694 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ac5a019d552488d82dd806b6d9831fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ac5a019d552488d82dd806b6d9831fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45629 } }
I20260928 22:53:16.214213 5694 leader_election.cc:302] T 00000000000000000000000000000000 P 9ac5a019d552488d82dd806b6d9831fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ac5a019d552488d82dd806b6d9831fa; no voters:
I20260928 22:53:16.214398 5694 leader_election.cc:288] T 00000000000000000000000000000000 P 9ac5a019d552488d82dd806b6d9831fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:53:16.214444 5699 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ac5a019d552488d82dd806b6d9831fa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:53:16.214613 5699 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ac5a019d552488d82dd806b6d9831fa [term 1 LEADER]: Becoming Leader. State: Replica: 9ac5a019d552488d82dd806b6d9831fa, State: Running, Role: LEADER
I20260928 22:53:16.214663 5694 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ac5a019d552488d82dd806b6d9831fa [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:53:16.214720 5699 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ac5a019d552488d82dd806b6d9831fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ac5a019d552488d82dd806b6d9831fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45629 } }
I20260928 22:53:16.215273 5700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ac5a019d552488d82dd806b6d9831fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ac5a019d552488d82dd806b6d9831fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ac5a019d552488d82dd806b6d9831fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45629 } } }
I20260928 22:53:16.215354 5700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ac5a019d552488d82dd806b6d9831fa [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:16.215545 5699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ac5a019d552488d82dd806b6d9831fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ac5a019d552488d82dd806b6d9831fa. Latest consensus state: current_term: 1 leader_uuid: "9ac5a019d552488d82dd806b6d9831fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ac5a019d552488d82dd806b6d9831fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45629 } } }
I20260928 22:53:16.215623 5699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ac5a019d552488d82dd806b6d9831fa [sys.catalog]: This master's current role is: LEADER
W20260928 22:53:16.216504 5715 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9ac5a019d552488d82dd806b6d9831fa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:53:16.216560 5715 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:53:16.216657 5708 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:53:16.216858 5708 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:53:16.219035 5708 catalog_manager.cc:1347] Generated new cluster ID: 67566165542d464e9a2dc8019128ee05
I20260928 22:53:16.219121 5708 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 22:53:16 dist-test-slave-56vw krb5kdc[5613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 22:53:16.223856 5698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:16.211095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.132:59035 (local address 127.0.0.1:45629)
0928 22:53:16.211311 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:16.211322 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:16.211362 (+ 40us) server_negotiation.cc:407] Connection header received
0928 22:53:16.212620 (+ 1258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:16.212629 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:16.212692 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:16.212879 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:16.213709 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:16.214902 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:16.216084 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:16.216704 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:16.217216 (+ 512us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:16.219813 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:16.219818 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:16.219818 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:16.219832 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:16.221654 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:16.222112 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:16.222115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:16.222115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:16.222627 (+ 512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:16.222901 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:16.222902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:16.222902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:16.223228 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:16.223425 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:16.223542 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 22:53:16.223661 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":118,"threads_started":1}
I20260928 22:53:16.224939 5708 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:53:16.225107 16870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:53:16.227182 5708 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:53:16.232261 5708 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ac5a019d552488d82dd806b6d9831fa: Generated new TSK 0
I20260928 22:53:16.232484 5708 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:53:16.233899 5698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:16.226731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.132:56873 (local address 127.0.0.1:45629)
0928 22:53:16.226774 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:16.226776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:16.226941 (+ 165us) server_negotiation.cc:407] Connection header received
0928 22:53:16.227024 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:16.227025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:16.227192 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:16.227253 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:16.227840 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:16.228371 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:16.229162 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:16.229403 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:16.229506 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:16.230785 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:16.230786 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:16.230786 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:16.230794 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:16.232728 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:16.233119 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:16.233120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:16.233121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:16.233188 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:16.233398 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:16.233399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:16.233399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:16.233531 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:16.233630 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:16.233740 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 22:53:16.233807 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:53:16.236135 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 5626
W20260928 22:53:16.240603 5611 connection.cc:535] client connection to 127.0.0.1:45629 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45629): Connection reset by peer (error 104)
2026-09-28T22:53:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1380 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635733329745-16870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:53:16 dist-test-slave-56vw krb5kdc[5725](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:53:16 dist-test-slave-56vw krb5kdc[5725](info): set up 2 sockets
Sep 28 22:53:16 dist-test-slave-56vw krb5kdc[5725](info): commencing operation
krb5kdc: starting...
W20260928 22:53:17.271371 16870 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Sep 28 22:53:17 dist-test-slave-56vw krb5kdc[5725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:53:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:53:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635733329745-16870-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635733329745-16870-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 22:53:17.394400 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36143
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635733329745-16870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:17.476449 5738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:17.476668 5738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:17.476691 5738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:17.477792 5738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:53:17.477829 5738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:17.477843 5738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:17.477856 5738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:53:17.477869 5738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:53:17.479403 5738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36143
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635733329745-16870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:17.479614 5738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:17.479786 5738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:17.484800 5743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:17.484802 5744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:17.484828 5746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:17.485040 5738 server_base.cc:1034] running on GCE node
I20260928 22:53:17.485289 5738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:17.485579 5738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:17.486768 5738 hybrid_clock.cc:648] HybridClock initialized: now 1790635997486767 us; error 47 us; skew 500 ppm
Sep 28 22:53:17 dist-test-slave-56vw krb5kdc[5725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:17.489581 5738 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 22:53:17.489981 5738 webserver.cc:460] Webserver started at http://127.0.0.1:34471/ using document root <none> and password file <none>
I20260928 22:53:17.490170 5738 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:17.490221 5738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:17.490311 5738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:17.491456 5738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "cf80e1df5a844e6e991db4230f9055d3"
format_stamp: "Formatted at 2026-09-28 22:53:17 on dist-test-slave-56vw"
server_key: "69ba22a5904f2cdc54cf32dd129bd339"
server_key_iv: "943f72508de5c32ae9724b4b6052857e"
server_key_version: "encryptionkey@0"
I20260928 22:53:17.491812 5738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "cf80e1df5a844e6e991db4230f9055d3"
format_stamp: "Formatted at 2026-09-28 22:53:17 on dist-test-slave-56vw"
server_key: "69ba22a5904f2cdc54cf32dd129bd339"
server_key_iv: "943f72508de5c32ae9724b4b6052857e"
server_key_version: "encryptionkey@0"
I20260928 22:53:17.493178 5738 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:53:17.493937 5753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:17.494112 5738 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:53:17.494189 5738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "cf80e1df5a844e6e991db4230f9055d3"
format_stamp: "Formatted at 2026-09-28 22:53:17 on dist-test-slave-56vw"
server_key: "69ba22a5904f2cdc54cf32dd129bd339"
server_key_iv: "943f72508de5c32ae9724b4b6052857e"
server_key_version: "encryptionkey@0"
I20260928 22:53:17.494325 5738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:17.508279 5738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:17.510715 5738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:17.510838 5738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:17.514897 5738 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36143
I20260928 22:53:17.514983 5805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36143 every 8 connection(s)
I20260928 22:53:17.515389 5738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:53:17.516333 5806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:17.518743 5806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf80e1df5a844e6e991db4230f9055d3: Bootstrap starting.
I20260928 22:53:17.519454 5806 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf80e1df5a844e6e991db4230f9055d3: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:17.519552 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 5738
I20260928 22:53:17.519680 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:53:17.519833 5806 log.cc:824] T 00000000000000000000000000000000 P cf80e1df5a844e6e991db4230f9055d3: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:17.519881 16870 external_mini_cluster.cc:1428] Setting key 4390088fba6506f67ee518f738b1f913
I20260928 22:53:17.520658 5806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf80e1df5a844e6e991db4230f9055d3: No bootstrap required, opened a new log
I20260928 22:53:17.523525 5806 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf80e1df5a844e6e991db4230f9055d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf80e1df5a844e6e991db4230f9055d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36143 } }
I20260928 22:53:17.523718 5806 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf80e1df5a844e6e991db4230f9055d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:17.523754 5806 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf80e1df5a844e6e991db4230f9055d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf80e1df5a844e6e991db4230f9055d3, State: Initialized, Role: FOLLOWER
I20260928 22:53:17.523883 5806 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf80e1df5a844e6e991db4230f9055d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf80e1df5a844e6e991db4230f9055d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36143 } }
I20260928 22:53:17.523977 5806 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf80e1df5a844e6e991db4230f9055d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:53:17.524003 5806 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf80e1df5a844e6e991db4230f9055d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:53:17.524078 5806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf80e1df5a844e6e991db4230f9055d3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:17.524747 5806 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf80e1df5a844e6e991db4230f9055d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf80e1df5a844e6e991db4230f9055d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36143 } }
I20260928 22:53:17.524864 5806 leader_election.cc:302] T 00000000000000000000000000000000 P cf80e1df5a844e6e991db4230f9055d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf80e1df5a844e6e991db4230f9055d3; no voters:
I20260928 22:53:17.525027 5806 leader_election.cc:288] T 00000000000000000000000000000000 P cf80e1df5a844e6e991db4230f9055d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:53:17.525205 5812 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf80e1df5a844e6e991db4230f9055d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:53:17.525314 5806 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf80e1df5a844e6e991db4230f9055d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:53:17.525383 5812 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf80e1df5a844e6e991db4230f9055d3 [term 1 LEADER]: Becoming Leader. State: Replica: cf80e1df5a844e6e991db4230f9055d3, State: Running, Role: LEADER
I20260928 22:53:17.525516 5812 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf80e1df5a844e6e991db4230f9055d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf80e1df5a844e6e991db4230f9055d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36143 } }
I20260928 22:53:17.525954 5811 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf80e1df5a844e6e991db4230f9055d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf80e1df5a844e6e991db4230f9055d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf80e1df5a844e6e991db4230f9055d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36143 } } }
I20260928 22:53:17.525970 5813 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf80e1df5a844e6e991db4230f9055d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf80e1df5a844e6e991db4230f9055d3. Latest consensus state: current_term: 1 leader_uuid: "cf80e1df5a844e6e991db4230f9055d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf80e1df5a844e6e991db4230f9055d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36143 } } }
I20260928 22:53:17.526060 5811 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf80e1df5a844e6e991db4230f9055d3 [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:17.526077 5813 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf80e1df5a844e6e991db4230f9055d3 [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:17.526351 5820 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:53:17.526374 5810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:17.520425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.132:44885 (local address 127.0.0.1:36143)
0928 22:53:17.520580 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:17.520595 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:17.520623 (+ 28us) server_negotiation.cc:407] Connection header received
0928 22:53:17.521147 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:17.521155 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:17.521209 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:17.521328 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:17.521562 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:17.522488 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:17.523924 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:17.524492 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:17.524584 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:17.525495 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:17.525500 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:17.525501 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:53:17.525694 (+ 193us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:17.525734 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:53:17.525749 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:17.525807 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:17.526010 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 22:53:17.526045 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":66,"threads_started":1}
I20260928 22:53:17.526850 5820 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:53:17.527493 16870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:53:17.528465 5820 catalog_manager.cc:1347] Generated new cluster ID: 8d8b9a6b289a46bcaf7ff73a1bb9b210
I20260928 22:53:17.528515 5820 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:53:17.532665 5810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:17.529082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.132:37445 (local address 127.0.0.1:36143)
0928 22:53:17.529116 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:17.529118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:17.529221 (+ 103us) server_negotiation.cc:407] Connection header received
0928 22:53:17.529252 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:17.529253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:17.529264 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:17.529330 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:17.529471 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:17.530218 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:17.531255 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:17.531556 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:17.531592 (+ 36us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:17.532404 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:17.532405 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:17.532406 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:53:17.532410 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:17.532427 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:53:17.532432 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:17.532500 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:17.532573 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 22:53:17.532585 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 22:53:17.553037 5820 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:53:17.553858 5820 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:53:17.561847 5820 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf80e1df5a844e6e991db4230f9055d3: Generated new TSK 0
I20260928 22:53:17.562093 5820 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:53:17.565924 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 5738
W20260928 22:53:17.570384 5723 connection.cc:535] client connection to 127.0.0.1:36143 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36143): Connection reset by peer (error 104)
2026-09-28T22:53:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635733329745-16870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:53:17 dist-test-slave-56vw krb5kdc[5837](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:53:17 dist-test-slave-56vw krb5kdc[5837](info): set up 2 sockets
Sep 28 22:53:17 dist-test-slave-56vw krb5kdc[5837](info): commencing operation
krb5kdc: starting...
W20260928 22:53:18.663329 16870 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Sep 28 22:53:18 dist-test-slave-56vw krb5kdc[5837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:53:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:53:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635733329745-16870-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635733329745-16870-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 22:53:18.791371 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36775
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635733329745-16870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:18.883711 5850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:18.883965 5850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:18.883991 5850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:18.885388 5850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:53:18.885447 5850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:18.885462 5850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:18.885479 5850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:53:18.885493 5850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:53:18.887339 5850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36775
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635733329745-16870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:18.887645 5850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:18.887918 5850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:18.893553 5856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:18.893559 5855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:18.893604 5850 server_base.cc:1034] running on GCE node
W20260928 22:53:18.893553 5858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:18.894002 5850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:18.894325 5850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:18.895542 5850 hybrid_clock.cc:648] HybridClock initialized: now 1790635998895516 us; error 48 us; skew 500 ppm
Sep 28 22:53:18 dist-test-slave-56vw krb5kdc[5837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790635998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:18.898288 5850 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 22:53:18.898805 5850 webserver.cc:460] Webserver started at http://127.0.0.1:39333/ using document root <none> and password file <none>
I20260928 22:53:18.899133 5850 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:18.899204 5850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:18.899364 5850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:18.900471 5850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "e6900d48537d44bd81580d7328189cfa"
format_stamp: "Formatted at 2026-09-28 22:53:18 on dist-test-slave-56vw"
server_key: "1eb31c4e18eb3f6cdbabc73d9e5a3a11"
server_key_iv: "e92799ed0d1da831ed3c3d56b612d736"
server_key_version: "encryptionkey@0"
I20260928 22:53:18.900863 5850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "e6900d48537d44bd81580d7328189cfa"
format_stamp: "Formatted at 2026-09-28 22:53:18 on dist-test-slave-56vw"
server_key: "1eb31c4e18eb3f6cdbabc73d9e5a3a11"
server_key_iv: "e92799ed0d1da831ed3c3d56b612d736"
server_key_version: "encryptionkey@0"
I20260928 22:53:18.902336 5850 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:53:18.903220 5865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:18.903457 5850 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:53:18.903573 5850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "e6900d48537d44bd81580d7328189cfa"
format_stamp: "Formatted at 2026-09-28 22:53:18 on dist-test-slave-56vw"
server_key: "1eb31c4e18eb3f6cdbabc73d9e5a3a11"
server_key_iv: "e92799ed0d1da831ed3c3d56b612d736"
server_key_version: "encryptionkey@0"
I20260928 22:53:18.903664 5850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:18.907172 5850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:18.909600 5850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:18.909770 5850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:18.914127 5850 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36775
I20260928 22:53:18.914163 5917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36775 every 8 connection(s)
I20260928 22:53:18.914537 5850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:53:18.915728 5918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:18.916808 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 5850
I20260928 22:53:18.916931 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:53:18.917131 16870 external_mini_cluster.cc:1428] Setting key 3499366432c11546f181ed17b470103b
I20260928 22:53:18.918622 5918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6900d48537d44bd81580d7328189cfa: Bootstrap starting.
I20260928 22:53:18.919414 5918 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6900d48537d44bd81580d7328189cfa: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:18.919718 5921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:18.917732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.132:42665 (local address 127.0.0.1:36775)
0928 22:53:18.917919 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:18.917929 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:18.917963 (+ 34us) server_negotiation.cc:407] Connection header received
0928 22:53:18.918381 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:18.918387 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:18.918413 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:18.918530 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:18.919275 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:18.919279 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:18.919279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:53:18.919283 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:18.919350 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:53:18.919362 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:18.919433 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:18.919523 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 22:53:18.919569 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":92,"threads_started":1}
I20260928 22:53:18.919759 5918 log.cc:824] T 00000000000000000000000000000000 P e6900d48537d44bd81580d7328189cfa: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:18.920537 5918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6900d48537d44bd81580d7328189cfa: No bootstrap required, opened a new log
I20260928 22:53:18.920778 16870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:53:18.923398 5921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:18.922325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.132:46921 (local address 127.0.0.1:36775)
0928 22:53:18.922363 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:18.922365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:18.922476 (+ 111us) server_negotiation.cc:407] Connection header received
0928 22:53:18.922504 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:18.922505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:18.922511 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:18.922582 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:18.923199 (+ 617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:18.923201 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:18.923201 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:53:18.923205 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:18.923220 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:53:18.923225 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:18.923294 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:18.923334 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 22:53:18.923347 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:53:18.923267 5918 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6900d48537d44bd81580d7328189cfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6900d48537d44bd81580d7328189cfa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36775 } }
I20260928 22:53:18.923471 5918 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6900d48537d44bd81580d7328189cfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:18.923507 5918 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6900d48537d44bd81580d7328189cfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6900d48537d44bd81580d7328189cfa, State: Initialized, Role: FOLLOWER
I20260928 22:53:18.923635 5918 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6900d48537d44bd81580d7328189cfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6900d48537d44bd81580d7328189cfa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36775 } }
I20260928 22:53:18.923740 5918 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6900d48537d44bd81580d7328189cfa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:53:18.923774 5918 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6900d48537d44bd81580d7328189cfa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:53:18.923831 5918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6900d48537d44bd81580d7328189cfa [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:18.924400 5918 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6900d48537d44bd81580d7328189cfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6900d48537d44bd81580d7328189cfa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36775 } }
I20260928 22:53:18.924535 5918 leader_election.cc:302] T 00000000000000000000000000000000 P e6900d48537d44bd81580d7328189cfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6900d48537d44bd81580d7328189cfa; no voters:
I20260928 22:53:18.924719 5918 leader_election.cc:288] T 00000000000000000000000000000000 P e6900d48537d44bd81580d7328189cfa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:53:18.924759 5928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6900d48537d44bd81580d7328189cfa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:53:18.924930 5928 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6900d48537d44bd81580d7328189cfa [term 1 LEADER]: Becoming Leader. State: Replica: e6900d48537d44bd81580d7328189cfa, State: Running, Role: LEADER
I20260928 22:53:18.924994 5918 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6900d48537d44bd81580d7328189cfa [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:53:18.925062 5928 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6900d48537d44bd81580d7328189cfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6900d48537d44bd81580d7328189cfa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36775 } }
I20260928 22:53:18.925498 5929 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6900d48537d44bd81580d7328189cfa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6900d48537d44bd81580d7328189cfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6900d48537d44bd81580d7328189cfa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36775 } } }
I20260928 22:53:18.925506 5930 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6900d48537d44bd81580d7328189cfa [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6900d48537d44bd81580d7328189cfa. Latest consensus state: current_term: 1 leader_uuid: "e6900d48537d44bd81580d7328189cfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6900d48537d44bd81580d7328189cfa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36775 } } }
I20260928 22:53:18.925611 5929 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6900d48537d44bd81580d7328189cfa [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:18.925616 5930 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6900d48537d44bd81580d7328189cfa [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:18.925848 5937 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:53:18.926363 5937 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:53:18.928118 5937 catalog_manager.cc:1347] Generated new cluster ID: ccb9028905d8415891e7410f868d8002
I20260928 22:53:18.928166 5937 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:53:18.935528 5937 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:53:18.935709 5937 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:53:18.941599 5937 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6900d48537d44bd81580d7328189cfa: Generated new TSK 0
I20260928 22:53:18.941797 5937 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:53:18.944648 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 5850
2026-09-28T22:53:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1379 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635733329745-16870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:53:19 dist-test-slave-56vw krb5kdc[5949](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:53:19 dist-test-slave-56vw krb5kdc[5949](info): set up 2 sockets
Sep 28 22:53:19 dist-test-slave-56vw krb5kdc[5949](info): commencing operation
krb5kdc: starting...
W20260928 22:53:20.015889 16870 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Sep 28 22:53:20 dist-test-slave-56vw krb5kdc[5949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:53:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:53:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.121.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
I20260928 22:53:20.143136 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:38667
--webserver_interface=127.16.121.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:38297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:38667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:20.225337 5962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:20.225566 5962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:20.225590 5962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:20.226824 5962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:53:20.226878 5962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:20.226893 5962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:20.226907 5962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:53:20.226919 5962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:53:20.228515 5962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:38297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:38667
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:38667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.121.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:20.228789 5962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:20.229066 5962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:20.234457 5967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:20.234460 5968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:20.234494 5962 server_base.cc:1034] running on GCE node
W20260928 22:53:20.234431 5970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:20.235008 5962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:20.235380 5962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:20.236591 5962 hybrid_clock.cc:648] HybridClock initialized: now 1790636000236576 us; error 49 us; skew 500 ppm
Sep 28 22:53:20 dist-test-slave-56vw krb5kdc[5949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636000, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:20.239439 5962 init.cc:377] Logged in from keytab as kudu/127.16.121.190@KRBTEST.COM (short username kudu)
I20260928 22:53:20.239938 5962 webserver.cc:460] Webserver started at http://127.16.121.190:34711/ using document root <none> and password file <none>
I20260928 22:53:20.240185 5962 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:20.240260 5962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:20.240411 5962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:20.241529 5962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "f96b7d986f044cbab95fc662609b5755"
format_stamp: "Formatted at 2026-09-28 22:53:20 on dist-test-slave-56vw"
server_key: "c72c6cd2ea3d035ddbf31b97f7c207d9"
server_key_iv: "4f5f7d3ca394209953784eb74b4f5e6f"
server_key_version: "encryptionkey@0"
I20260928 22:53:20.241909 5962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "f96b7d986f044cbab95fc662609b5755"
format_stamp: "Formatted at 2026-09-28 22:53:20 on dist-test-slave-56vw"
server_key: "c72c6cd2ea3d035ddbf31b97f7c207d9"
server_key_iv: "4f5f7d3ca394209953784eb74b4f5e6f"
server_key_version: "encryptionkey@0"
I20260928 22:53:20.243407 5962 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260928 22:53:20.244292 5977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:20.244491 5962 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:53:20.244602 5962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "f96b7d986f044cbab95fc662609b5755"
format_stamp: "Formatted at 2026-09-28 22:53:20 on dist-test-slave-56vw"
server_key: "c72c6cd2ea3d035ddbf31b97f7c207d9"
server_key_iv: "4f5f7d3ca394209953784eb74b4f5e6f"
server_key_version: "encryptionkey@0"
I20260928 22:53:20.244680 5962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:20.254069 5962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:20.256534 5962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:20.256705 5962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:20.261323 5962 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.190:38667
I20260928 22:53:20.261358 6029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.190:38667 every 8 connection(s)
I20260928 22:53:20.261802 5962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:53:20.262713 6030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:20.265237 6030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f96b7d986f044cbab95fc662609b5755: Bootstrap starting.
I20260928 22:53:20.265859 6030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f96b7d986f044cbab95fc662609b5755: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:20.266155 6030 log.cc:824] T 00000000000000000000000000000000 P f96b7d986f044cbab95fc662609b5755: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:20.267010 6030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f96b7d986f044cbab95fc662609b5755: No bootstrap required, opened a new log
I20260928 22:53:20.268285 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 5962
I20260928 22:53:20.268436 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:53:20.268649 16870 external_mini_cluster.cc:1428] Setting key ed0646f8c0172977f1d931bddde82df3
I20260928 22:53:20.268975 6030 raft_consensus.cc:348] T 00000000000000000000000000000000 P f96b7d986f044cbab95fc662609b5755 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f96b7d986f044cbab95fc662609b5755" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 38667 } }
I20260928 22:53:20.269208 6030 raft_consensus.cc:374] T 00000000000000000000000000000000 P f96b7d986f044cbab95fc662609b5755 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:20.269274 6030 raft_consensus.cc:729] T 00000000000000000000000000000000 P f96b7d986f044cbab95fc662609b5755 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f96b7d986f044cbab95fc662609b5755, State: Initialized, Role: FOLLOWER
I20260928 22:53:20.269475 6030 consensus_queue.cc:260] T 00000000000000000000000000000000 P f96b7d986f044cbab95fc662609b5755 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f96b7d986f044cbab95fc662609b5755" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 38667 } }
I20260928 22:53:20.269595 6030 raft_consensus.cc:388] T 00000000000000000000000000000000 P f96b7d986f044cbab95fc662609b5755 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:53:20.269631 6030 raft_consensus.cc:482] T 00000000000000000000000000000000 P f96b7d986f044cbab95fc662609b5755 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:53:20.269685 6030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f96b7d986f044cbab95fc662609b5755 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:20.270787 6030 raft_consensus.cc:504] T 00000000000000000000000000000000 P f96b7d986f044cbab95fc662609b5755 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f96b7d986f044cbab95fc662609b5755" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 38667 } }
I20260928 22:53:20.270898 6030 leader_election.cc:302] T 00000000000000000000000000000000 P f96b7d986f044cbab95fc662609b5755 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f96b7d986f044cbab95fc662609b5755; no voters:
I20260928 22:53:20.271076 6030 leader_election.cc:288] T 00000000000000000000000000000000 P f96b7d986f044cbab95fc662609b5755 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:53:20.271162 6035 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f96b7d986f044cbab95fc662609b5755 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:53:20.271334 6035 raft_consensus.cc:686] T 00000000000000000000000000000000 P f96b7d986f044cbab95fc662609b5755 [term 1 LEADER]: Becoming Leader. State: Replica: f96b7d986f044cbab95fc662609b5755, State: Running, Role: LEADER
I20260928 22:53:20.271404 6030 sys_catalog.cc:564] T 00000000000000000000000000000000 P f96b7d986f044cbab95fc662609b5755 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:53:20.271493 6035 consensus_queue.cc:237] T 00000000000000000000000000000000 P f96b7d986f044cbab95fc662609b5755 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f96b7d986f044cbab95fc662609b5755" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 38667 } }
I20260928 22:53:20.271868 6035 sys_catalog.cc:455] T 00000000000000000000000000000000 P f96b7d986f044cbab95fc662609b5755 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f96b7d986f044cbab95fc662609b5755. Latest consensus state: current_term: 1 leader_uuid: "f96b7d986f044cbab95fc662609b5755" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f96b7d986f044cbab95fc662609b5755" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 38667 } } }
I20260928 22:53:20.271883 6037 sys_catalog.cc:455] T 00000000000000000000000000000000 P f96b7d986f044cbab95fc662609b5755 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f96b7d986f044cbab95fc662609b5755" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f96b7d986f044cbab95fc662609b5755" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 38667 } } }
I20260928 22:53:20.271998 6035 sys_catalog.cc:458] T 00000000000000000000000000000000 P f96b7d986f044cbab95fc662609b5755 [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:20.272083 6037 sys_catalog.cc:458] T 00000000000000000000000000000000 P f96b7d986f044cbab95fc662609b5755 [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:20.272351 6044 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:53:20.272936 6044 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:53:20.274833 6044 catalog_manager.cc:1347] Generated new cluster ID: 4541a21f74f44c159b6ba88d7df0575b
I20260928 22:53:20.274899 6044 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 22:53:20 dist-test-slave-56vw krb5kdc[5949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:53:20.281598 6034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:20.269388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54231 (local address 127.16.121.190:38667)
0928 22:53:20.269620 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:20.269631 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:20.269669 (+ 38us) server_negotiation.cc:407] Connection header received
0928 22:53:20.269807 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:20.269815 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:20.269875 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:20.270036 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:20.270710 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:20.272157 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:20.273337 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:20.274000 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:20.274110 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:20.277204 (+ 3094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:20.277209 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:20.277209 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:20.277223 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:20.279446 (+ 2223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:20.279922 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:20.279926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:20.279926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:20.280066 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:20.280324 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:20.280325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:20.280326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:20.280698 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:20.280899 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:20.281006 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 22:53:20.281125 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":128,"threads_started":1}
I20260928 22:53:20.282754 16870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:53:20.292281 6034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:20.284627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35771 (local address 127.16.121.190:38667)
0928 22:53:20.284675 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:20.284677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:20.284686 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:53:20.284710 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:20.284711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:20.284722 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:20.284795 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:20.285412 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:20.286286 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:20.287335 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:20.287727 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:20.287807 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:20.289186 (+ 1379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:20.289188 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:20.289188 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:20.289195 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:20.290889 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:20.291365 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:20.291366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:20.291367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:20.291430 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:20.291664 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:20.291665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:20.291666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:20.291807 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:20.291935 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:20.292096 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 22:53:20.292168 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":9,"server-negotiator.queue_time_us":27}
I20260928 22:53:20.305725 6044 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:53:20.306433 6044 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:53:20.312712 6044 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f96b7d986f044cbab95fc662609b5755: Generated new TSK 0
I20260928 22:53:20.312933 6044 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:53:20.321420 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 5962
W20260928 22:53:20.325722 5947 connection.cc:535] client connection to 127.16.121.190:38667 recv error: Network error: failed to read from TLS socket (remote: 127.16.121.190:38667): Connection reset by peer (error 104)
2026-09-28T22:53:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1375 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635733329745-16870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:53:20 dist-test-slave-56vw krb5kdc[6061](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:53:20 dist-test-slave-56vw krb5kdc[6061](info): set up 2 sockets
Sep 28 22:53:20 dist-test-slave-56vw krb5kdc[6061](info): commencing operation
krb5kdc: starting...
W20260928 22:53:21.391620 16870 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Sep 28 22:53:21 dist-test-slave-56vw krb5kdc[6061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:53:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:53:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.121.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
I20260928 22:53:21.518733 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:36621
--webserver_interface=127.16.121.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:43733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:36621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:21.601893 6074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:21.602114 6074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:21.602139 6074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:21.603402 6074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:53:21.603451 6074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:21.603466 6074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:21.603478 6074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:53:21.603492 6074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:53:21.604954 6074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:43733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:36621
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:36621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.121.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:21.605149 6074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:21.605325 6074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:21.610672 6079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:21.610704 6080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:21.610714 6082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:21.610750 6074 server_base.cc:1034] running on GCE node
I20260928 22:53:21.611320 6074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:21.611671 6074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:21.612870 6074 hybrid_clock.cc:648] HybridClock initialized: now 1790636001612860 us; error 44 us; skew 500 ppm
Sep 28 22:53:21 dist-test-slave-56vw krb5kdc[6061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636001, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:21.621078 6074 init.cc:377] Logged in from keytab as kudu/127.16.121.190@KRBTEST.COM (short username kudu)
I20260928 22:53:21.621536 6074 webserver.cc:460] Webserver started at http://127.16.121.190:43199/ using document root <none> and password file <none>
I20260928 22:53:21.621729 6074 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:21.621775 6074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:21.621870 6074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:21.622967 6074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "e9456112cb9245a79416ec920ba99776"
format_stamp: "Formatted at 2026-09-28 22:53:21 on dist-test-slave-56vw"
server_key: "206cbd7e16f14ab44c747b73c90790be"
server_key_iv: "7fe77c41caa16ba27ce27a42a6fd81d3"
server_key_version: "encryptionkey@0"
I20260928 22:53:21.623395 6074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "e9456112cb9245a79416ec920ba99776"
format_stamp: "Formatted at 2026-09-28 22:53:21 on dist-test-slave-56vw"
server_key: "206cbd7e16f14ab44c747b73c90790be"
server_key_iv: "7fe77c41caa16ba27ce27a42a6fd81d3"
server_key_version: "encryptionkey@0"
I20260928 22:53:21.624900 6074 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 22:53:21.625905 6089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:21.626330 6074 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:53:21.626456 6074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "e9456112cb9245a79416ec920ba99776"
format_stamp: "Formatted at 2026-09-28 22:53:21 on dist-test-slave-56vw"
server_key: "206cbd7e16f14ab44c747b73c90790be"
server_key_iv: "7fe77c41caa16ba27ce27a42a6fd81d3"
server_key_version: "encryptionkey@0"
I20260928 22:53:21.626547 6074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:21.651422 6074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:21.653770 6074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:21.653944 6074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:21.658201 6074 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.190:36621
I20260928 22:53:21.658288 6141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.190:36621 every 8 connection(s)
I20260928 22:53:21.658644 6074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:53:21.659642 6142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:21.661991 6142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9456112cb9245a79416ec920ba99776: Bootstrap starting.
I20260928 22:53:21.662586 6142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9456112cb9245a79416ec920ba99776: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:21.662889 6142 log.cc:824] T 00000000000000000000000000000000 P e9456112cb9245a79416ec920ba99776: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:21.663861 6142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9456112cb9245a79416ec920ba99776: No bootstrap required, opened a new log
I20260928 22:53:21.664270 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 6074
I20260928 22:53:21.664374 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:53:21.664592 16870 external_mini_cluster.cc:1428] Setting key 0a4697543cdb609e665e5159e32dba94
I20260928 22:53:21.666464 6142 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9456112cb9245a79416ec920ba99776 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9456112cb9245a79416ec920ba99776" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 36621 } }
I20260928 22:53:21.666631 6142 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9456112cb9245a79416ec920ba99776 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:21.666682 6142 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9456112cb9245a79416ec920ba99776 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9456112cb9245a79416ec920ba99776, State: Initialized, Role: FOLLOWER
I20260928 22:53:21.666808 6142 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9456112cb9245a79416ec920ba99776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9456112cb9245a79416ec920ba99776" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 36621 } }
I20260928 22:53:21.666898 6142 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9456112cb9245a79416ec920ba99776 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:53:21.666941 6142 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9456112cb9245a79416ec920ba99776 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:53:21.666992 6142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9456112cb9245a79416ec920ba99776 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:21.667632 6142 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9456112cb9245a79416ec920ba99776 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9456112cb9245a79416ec920ba99776" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 36621 } }
I20260928 22:53:21.667768 6142 leader_election.cc:302] T 00000000000000000000000000000000 P e9456112cb9245a79416ec920ba99776 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9456112cb9245a79416ec920ba99776; no voters:
I20260928 22:53:21.667987 6142 leader_election.cc:288] T 00000000000000000000000000000000 P e9456112cb9245a79416ec920ba99776 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:53:21.668010 6147 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9456112cb9245a79416ec920ba99776 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:53:21.668126 6147 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9456112cb9245a79416ec920ba99776 [term 1 LEADER]: Becoming Leader. State: Replica: e9456112cb9245a79416ec920ba99776, State: Running, Role: LEADER
I20260928 22:53:21.668219 6147 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9456112cb9245a79416ec920ba99776 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9456112cb9245a79416ec920ba99776" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 36621 } }
I20260928 22:53:21.668282 6142 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9456112cb9245a79416ec920ba99776 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:53:21.668520 6148 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9456112cb9245a79416ec920ba99776 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9456112cb9245a79416ec920ba99776" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9456112cb9245a79416ec920ba99776" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 36621 } } }
I20260928 22:53:21.668723 6148 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9456112cb9245a79416ec920ba99776 [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:21.668664 6149 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9456112cb9245a79416ec920ba99776 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9456112cb9245a79416ec920ba99776. Latest consensus state: current_term: 1 leader_uuid: "e9456112cb9245a79416ec920ba99776" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9456112cb9245a79416ec920ba99776" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 36621 } } }
I20260928 22:53:21.669008 6149 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9456112cb9245a79416ec920ba99776 [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:21.669395 6154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:53:21.669893 6154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:53:21.671334 6146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:21.665116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39603 (local address 127.16.121.190:36621)
0928 22:53:21.665322 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:21.665329 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:21.665356 (+ 27us) server_negotiation.cc:407] Connection header received
0928 22:53:21.665809 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:21.665816 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:21.665866 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:21.665983 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:21.666340 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:21.667664 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:21.668833 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:21.669479 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:21.669562 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:21.670566 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:21.670570 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:21.670571 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:53:21.670781 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:21.670819 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:53:21.670835 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:21.670921 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:21.671089 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 22:53:21.671117 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":94,"threads_started":1}
I20260928 22:53:21.672513 16870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:53:21.675601 6154 catalog_manager.cc:1347] Generated new cluster ID: c780b100bc1d4911ad210a5c7289af37
I20260928 22:53:21.675657 6154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:53:21.677860 6146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:21.674094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42539 (local address 127.16.121.190:36621)
0928 22:53:21.674133 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:21.674135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:21.674205 (+ 70us) server_negotiation.cc:407] Connection header received
0928 22:53:21.674268 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:21.674268 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:21.674275 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:21.674320 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:21.674451 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:21.674928 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:21.676004 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:21.676719 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:21.676792 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:21.677589 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:21.677590 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:21.677590 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:53:21.677594 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:21.677608 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:53:21.677613 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:21.677667 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:21.677774 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 22:53:21.677785 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 22:53:21.685684 6154 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:53:21.686322 6154 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:53:21.692212 6154 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9456112cb9245a79416ec920ba99776: Generated new TSK 0
I20260928 22:53:21.692505 6154 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:53:21.702207 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 6074
W20260928 22:53:21.706497 6059 connection.cc:535] client connection to 127.16.121.190:36621 recv error: Network error: failed to read from TLS socket (remote: 127.16.121.190:36621): Connection reset by peer (error 104)
2026-09-28T22:53:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1380 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635733329745-16870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:53:21 dist-test-slave-56vw krb5kdc[6173](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:53:21 dist-test-slave-56vw krb5kdc[6173](info): set up 2 sockets
Sep 28 22:53:21 dist-test-slave-56vw krb5kdc[6173](info): commencing operation
krb5kdc: starting...
W20260928 22:53:22.751366 16870 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Sep 28 22:53:22 dist-test-slave-56vw krb5kdc[6173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:53:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:53:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.121.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
I20260928 22:53:22.879854 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:46191
--webserver_interface=127.16.121.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:34187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:46191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:22.961808 6186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:22.962087 6186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:22.962128 6186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:22.963407 6186 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:53:22.963491 6186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:22.963515 6186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:22.963541 6186 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:53:22.963573 6186 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:53:22.965119 6186 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:34187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:46191
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:46191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.121.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:22.965418 6186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:22.965662 6186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 22:53:22.970793 6186 server_base.cc:1034] running on GCE node
W20260928 22:53:22.970793 6191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:22.970803 6192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:22.970820 6194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:22.971266 6186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:22.971590 6186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:22.972771 6186 hybrid_clock.cc:648] HybridClock initialized: now 1790636002972767 us; error 40 us; skew 500 ppm
Sep 28 22:53:22 dist-test-slave-56vw krb5kdc[6173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636002, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:22.975325 6186 init.cc:377] Logged in from keytab as kudu/127.16.121.190@KRBTEST.COM (short username kudu)
I20260928 22:53:22.975875 6186 webserver.cc:460] Webserver started at http://127.16.121.190:37793/ using document root <none> and password file <none>
I20260928 22:53:22.976122 6186 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:22.976195 6186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:22.976347 6186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:22.977465 6186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "8383b02c52f04ed2af3e71ef6700a1b4"
format_stamp: "Formatted at 2026-09-28 22:53:22 on dist-test-slave-56vw"
server_key: "bec825713c3e00e7890373d88cc65453"
server_key_iv: "590ede68de28af4bb707b7e4f31dbece"
server_key_version: "encryptionkey@0"
I20260928 22:53:22.977850 6186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "8383b02c52f04ed2af3e71ef6700a1b4"
format_stamp: "Formatted at 2026-09-28 22:53:22 on dist-test-slave-56vw"
server_key: "bec825713c3e00e7890373d88cc65453"
server_key_iv: "590ede68de28af4bb707b7e4f31dbece"
server_key_version: "encryptionkey@0"
I20260928 22:53:22.979394 6186 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:53:22.980201 6201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:22.980405 6186 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:53:22.980518 6186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "8383b02c52f04ed2af3e71ef6700a1b4"
format_stamp: "Formatted at 2026-09-28 22:53:22 on dist-test-slave-56vw"
server_key: "bec825713c3e00e7890373d88cc65453"
server_key_iv: "590ede68de28af4bb707b7e4f31dbece"
server_key_version: "encryptionkey@0"
I20260928 22:53:22.980598 6186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:22.984143 6186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:22.986233 6186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:22.986414 6186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:22.990813 6186 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.190:46191
I20260928 22:53:22.990844 6253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.190:46191 every 8 connection(s)
I20260928 22:53:22.991271 6186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:53:22.992311 6254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:22.994544 6254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8383b02c52f04ed2af3e71ef6700a1b4: Bootstrap starting.
I20260928 22:53:22.994879 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 6186
I20260928 22:53:22.995015 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:53:22.995182 6254 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8383b02c52f04ed2af3e71ef6700a1b4: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:22.995237 16870 external_mini_cluster.cc:1428] Setting key 94e20f5b16142acda32959f2a6ec7e79
I20260928 22:53:22.995530 6254 log.cc:824] T 00000000000000000000000000000000 P 8383b02c52f04ed2af3e71ef6700a1b4: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:22.996335 6254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8383b02c52f04ed2af3e71ef6700a1b4: No bootstrap required, opened a new log
I20260928 22:53:22.998342 6258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:22.995836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45831 (local address 127.16.121.190:46191)
0928 22:53:22.996076 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:22.996087 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:22.996148 (+ 61us) server_negotiation.cc:407] Connection header received
0928 22:53:22.996772 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:22.996781 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:22.996818 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:22.996975 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:22.997700 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:22.997704 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:22.997705 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:53:22.997709 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:22.997756 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:53:22.997772 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:22.997862 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:22.997905 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 22:53:22.997945 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":97,"threads_started":1}
I20260928 22:53:22.998435 6254 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8383b02c52f04ed2af3e71ef6700a1b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8383b02c52f04ed2af3e71ef6700a1b4" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 46191 } }
I20260928 22:53:22.998600 6254 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8383b02c52f04ed2af3e71ef6700a1b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:22.998629 6254 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8383b02c52f04ed2af3e71ef6700a1b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8383b02c52f04ed2af3e71ef6700a1b4, State: Initialized, Role: FOLLOWER
I20260928 22:53:22.998761 6254 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8383b02c52f04ed2af3e71ef6700a1b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8383b02c52f04ed2af3e71ef6700a1b4" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 46191 } }
I20260928 22:53:22.998879 6254 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8383b02c52f04ed2af3e71ef6700a1b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:53:22.998925 6254 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8383b02c52f04ed2af3e71ef6700a1b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:53:22.998970 6254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8383b02c52f04ed2af3e71ef6700a1b4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:22.999507 16870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:53:22.999768 6254 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8383b02c52f04ed2af3e71ef6700a1b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8383b02c52f04ed2af3e71ef6700a1b4" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 46191 } }
I20260928 22:53:22.999893 6254 leader_election.cc:302] T 00000000000000000000000000000000 P 8383b02c52f04ed2af3e71ef6700a1b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8383b02c52f04ed2af3e71ef6700a1b4; no voters:
I20260928 22:53:23.000057 6254 leader_election.cc:288] T 00000000000000000000000000000000 P 8383b02c52f04ed2af3e71ef6700a1b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:53:23.000116 6259 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8383b02c52f04ed2af3e71ef6700a1b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:53:23.000278 6259 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8383b02c52f04ed2af3e71ef6700a1b4 [term 1 LEADER]: Becoming Leader. State: Replica: 8383b02c52f04ed2af3e71ef6700a1b4, State: Running, Role: LEADER
I20260928 22:53:23.000313 6254 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8383b02c52f04ed2af3e71ef6700a1b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:53:23.000370 6259 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8383b02c52f04ed2af3e71ef6700a1b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8383b02c52f04ed2af3e71ef6700a1b4" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 46191 } }
I20260928 22:53:23.000692 6259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8383b02c52f04ed2af3e71ef6700a1b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8383b02c52f04ed2af3e71ef6700a1b4. Latest consensus state: current_term: 1 leader_uuid: "8383b02c52f04ed2af3e71ef6700a1b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8383b02c52f04ed2af3e71ef6700a1b4" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 46191 } } }
I20260928 22:53:23.000787 6259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8383b02c52f04ed2af3e71ef6700a1b4 [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:23.000712 6260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8383b02c52f04ed2af3e71ef6700a1b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8383b02c52f04ed2af3e71ef6700a1b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8383b02c52f04ed2af3e71ef6700a1b4" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 46191 } } }
I20260928 22:53:23.000934 6260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8383b02c52f04ed2af3e71ef6700a1b4 [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:23.001230 6270 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:53:23.001988 6270 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:53:23.003059 6258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:23.001441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55401 (local address 127.16.121.190:46191)
0928 22:53:23.001479 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:23.001481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:23.001918 (+ 437us) server_negotiation.cc:407] Connection header received
0928 22:53:23.002026 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:23.002026 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:23.002032 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:23.002064 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:23.002784 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:23.002785 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:23.002785 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:53:23.002789 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:23.002803 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:53:23.002807 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:23.002860 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:23.002955 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 22:53:23.002966 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:53:23.008168 6270 catalog_manager.cc:1347] Generated new cluster ID: af92361802fe4f62bf7999829c7279aa
I20260928 22:53:23.008221 6270 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:53:23.023142 6270 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:53:23.023348 6270 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:53:23.030879 6270 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8383b02c52f04ed2af3e71ef6700a1b4: Generated new TSK 0
I20260928 22:53:23.031123 6270 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:53:23.032907 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 6186
2026-09-28T22:53:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1331 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635733329745-16870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:53:23 dist-test-slave-56vw krb5kdc[6285](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:53:23 dist-test-slave-56vw krb5kdc[6285](info): set up 2 sockets
Sep 28 22:53:23 dist-test-slave-56vw krb5kdc[6285](info): commencing operation
krb5kdc: starting...
W20260928 22:53:24.107357 16870 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Sep 28 22:53:24 dist-test-slave-56vw krb5kdc[6285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:53:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:53:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.121.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
I20260928 22:53:24.234836 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:46333
--webserver_interface=127.16.121.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:45313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:46333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:24.320998 6298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:24.321233 6298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:24.321261 6298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:24.322530 6298 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:53:24.322583 6298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:24.322598 6298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:24.322611 6298 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:53:24.322623 6298 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:53:24.324294 6298 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:45313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:46333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:46333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.121.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:24.324587 6298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:24.324842 6298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:24.330236 6304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:24.330291 6303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:24.330399 6306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:24.330432 6298 server_base.cc:1034] running on GCE node
I20260928 22:53:24.330773 6298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:24.331030 6298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:24.332304 6298 hybrid_clock.cc:648] HybridClock initialized: now 1790636004332265 us; error 73 us; skew 500 ppm
Sep 28 22:53:24 dist-test-slave-56vw krb5kdc[6285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636004, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:24.334893 6298 init.cc:377] Logged in from keytab as kudu/127.16.121.190@KRBTEST.COM (short username kudu)
I20260928 22:53:24.335369 6298 webserver.cc:460] Webserver started at http://127.16.121.190:37315/ using document root <none> and password file <none>
I20260928 22:53:24.335566 6298 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:24.335608 6298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:24.335695 6298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:24.336702 6298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "cbef9bd6856d4f82b5ef83eb03c739f6"
format_stamp: "Formatted at 2026-09-28 22:53:24 on dist-test-slave-56vw"
server_key: "f4d4c672283d95afd54c08cc11c0b2ae"
server_key_iv: "fba147df0dc087ced02270be33d61740"
server_key_version: "encryptionkey@0"
I20260928 22:53:24.337044 6298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "cbef9bd6856d4f82b5ef83eb03c739f6"
format_stamp: "Formatted at 2026-09-28 22:53:24 on dist-test-slave-56vw"
server_key: "f4d4c672283d95afd54c08cc11c0b2ae"
server_key_iv: "fba147df0dc087ced02270be33d61740"
server_key_version: "encryptionkey@0"
I20260928 22:53:24.338373 6298 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 22:53:24.339110 6313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:24.339340 6298 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:53:24.339437 6298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "cbef9bd6856d4f82b5ef83eb03c739f6"
format_stamp: "Formatted at 2026-09-28 22:53:24 on dist-test-slave-56vw"
server_key: "f4d4c672283d95afd54c08cc11c0b2ae"
server_key_iv: "fba147df0dc087ced02270be33d61740"
server_key_version: "encryptionkey@0"
I20260928 22:53:24.339540 6298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:24.352177 6298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:24.354637 6298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:24.354807 6298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:24.358801 6298 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.190:46333
I20260928 22:53:24.358892 6365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.190:46333 every 8 connection(s)
I20260928 22:53:24.359278 6298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:53:24.360138 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 6298
I20260928 22:53:24.360271 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:53:24.360479 6366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:24.360513 16870 external_mini_cluster.cc:1428] Setting key defeec580217bf85ff6622e63bea9884
I20260928 22:53:24.364140 6366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbef9bd6856d4f82b5ef83eb03c739f6: Bootstrap starting.
I20260928 22:53:24.364935 6366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbef9bd6856d4f82b5ef83eb03c739f6: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:24.365271 6366 log.cc:824] T 00000000000000000000000000000000 P cbef9bd6856d4f82b5ef83eb03c739f6: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:24.366053 6366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbef9bd6856d4f82b5ef83eb03c739f6: No bootstrap required, opened a new log
Sep 28 22:53:24 dist-test-slave-56vw krb5kdc[6285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:53:24.369426 6366 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbef9bd6856d4f82b5ef83eb03c739f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbef9bd6856d4f82b5ef83eb03c739f6" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 46333 } }
I20260928 22:53:24.369634 6366 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbef9bd6856d4f82b5ef83eb03c739f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:24.369675 6366 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbef9bd6856d4f82b5ef83eb03c739f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbef9bd6856d4f82b5ef83eb03c739f6, State: Initialized, Role: FOLLOWER
I20260928 22:53:24.369865 6366 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbef9bd6856d4f82b5ef83eb03c739f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbef9bd6856d4f82b5ef83eb03c739f6" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 46333 } }
I20260928 22:53:24.369980 6366 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbef9bd6856d4f82b5ef83eb03c739f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:53:24.370019 6366 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbef9bd6856d4f82b5ef83eb03c739f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:53:24.370083 6366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbef9bd6856d4f82b5ef83eb03c739f6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:24.370911 6366 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbef9bd6856d4f82b5ef83eb03c739f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbef9bd6856d4f82b5ef83eb03c739f6" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 46333 } }
I20260928 22:53:24.371074 6366 leader_election.cc:302] T 00000000000000000000000000000000 P cbef9bd6856d4f82b5ef83eb03c739f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbef9bd6856d4f82b5ef83eb03c739f6; no voters:
I20260928 22:53:24.371294 6366 leader_election.cc:288] T 00000000000000000000000000000000 P cbef9bd6856d4f82b5ef83eb03c739f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:53:24.371351 6371 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbef9bd6856d4f82b5ef83eb03c739f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:53:24.371510 6371 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbef9bd6856d4f82b5ef83eb03c739f6 [term 1 LEADER]: Becoming Leader. State: Replica: cbef9bd6856d4f82b5ef83eb03c739f6, State: Running, Role: LEADER
I20260928 22:53:24.371596 6366 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbef9bd6856d4f82b5ef83eb03c739f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:53:24.371677 6371 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbef9bd6856d4f82b5ef83eb03c739f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbef9bd6856d4f82b5ef83eb03c739f6" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 46333 } }
I20260928 22:53:24.372145 6373 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbef9bd6856d4f82b5ef83eb03c739f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbef9bd6856d4f82b5ef83eb03c739f6. Latest consensus state: current_term: 1 leader_uuid: "cbef9bd6856d4f82b5ef83eb03c739f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbef9bd6856d4f82b5ef83eb03c739f6" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 46333 } } }
I20260928 22:53:24.372148 6372 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbef9bd6856d4f82b5ef83eb03c739f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbef9bd6856d4f82b5ef83eb03c739f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbef9bd6856d4f82b5ef83eb03c739f6" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 46333 } } }
I20260928 22:53:24.372300 6373 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbef9bd6856d4f82b5ef83eb03c739f6 [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:24.372308 6372 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbef9bd6856d4f82b5ef83eb03c739f6 [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:24.373240 6369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:24.361100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41081 (local address 127.16.121.190:46333)
0928 22:53:24.361326 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:24.361335 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:24.361368 (+ 33us) server_negotiation.cc:407] Connection header received
0928 22:53:24.362138 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:24.362147 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:24.362217 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:24.362374 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:24.363009 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:24.364631 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:24.365569 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:24.365990 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:24.366129 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:24.368850 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:24.368855 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:24.368855 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:24.368873 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:24.371197 (+ 2324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:24.371715 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:24.371718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:24.371719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:24.371825 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:24.372077 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:24.372079 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:24.372079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:24.372489 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:24.372688 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:24.372780 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 22:53:24.372969 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":79,"threads_started":1}
W20260928 22:53:24.373654 6386 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cbef9bd6856d4f82b5ef83eb03c739f6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:53:24.373710 6386 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:53:24.373780 6380 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:53:24.373958 6380 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:53:24.374632 16870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:53:24.375682 6380 catalog_manager.cc:1347] Generated new cluster ID: 8a440602b9744a66a03ec100c764cb63
I20260928 22:53:24.375732 6380 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:53:24.387846 6369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:24.379469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40925 (local address 127.16.121.190:46333)
0928 22:53:24.379508 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:24.379510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:24.379727 (+ 217us) server_negotiation.cc:407] Connection header received
0928 22:53:24.379807 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:24.379807 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:24.379817 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:24.379881 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:24.380671 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:24.381425 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:24.382600 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:24.382910 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:24.382967 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:24.384674 (+ 1707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:24.384675 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:24.384676 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:24.384682 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:24.386408 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:24.386856 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:24.386857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:24.386858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:24.386943 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:24.387232 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:24.387233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:24.387233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:24.387399 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:24.387492 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:24.387664 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 22:53:24.387744 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:53:24.398871 6380 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:53:24.399794 6380 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:53:24.405855 6380 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbef9bd6856d4f82b5ef83eb03c739f6: Generated new TSK 0
I20260928 22:53:24.406116 6380 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:53:24.410471 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 6298
2026-09-28T22:53:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1377 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635733329745-16870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:53:24 dist-test-slave-56vw krb5kdc[6397](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:53:24 dist-test-slave-56vw krb5kdc[6397](info): set up 2 sockets
Sep 28 22:53:24 dist-test-slave-56vw krb5kdc[6397](info): commencing operation
krb5kdc: starting...
W20260928 22:53:25.463410 16870 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Sep 28 22:53:25 dist-test-slave-56vw krb5kdc[6397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:53:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:53:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.121.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
I20260928 22:53:25.592247 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:42455
--webserver_interface=127.16.121.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:35631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:42455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:25.679877 6410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:25.680148 6410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:25.680188 6410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:25.681383 6410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:53:25.681459 6410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:25.681484 6410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:25.681512 6410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:53:25.681543 6410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:53:25.683105 6410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:35631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:42455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:42455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.121.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:25.683398 6410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:25.683638 6410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:25.689267 6418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:25.689375 6416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:25.689267 6415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:25.689750 6410 server_base.cc:1034] running on GCE node
I20260928 22:53:25.690037 6410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:25.690358 6410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:25.691540 6410 hybrid_clock.cc:648] HybridClock initialized: now 1790636005691507 us; error 66 us; skew 500 ppm
Sep 28 22:53:25 dist-test-slave-56vw krb5kdc[6397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636005, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:25.694178 6410 init.cc:377] Logged in from keytab as kudu/127.16.121.190@KRBTEST.COM (short username kudu)
I20260928 22:53:25.694667 6410 webserver.cc:460] Webserver started at http://127.16.121.190:44959/ using document root <none> and password file <none>
I20260928 22:53:25.694918 6410 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:25.694991 6410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:25.695170 6410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:25.696273 6410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "24fa558b3a1c425bb75f87e592beec33"
format_stamp: "Formatted at 2026-09-28 22:53:25 on dist-test-slave-56vw"
server_key: "2d957263cf6fe2d0421cb40cab3a7595"
server_key_iv: "eca874c74ab4f159ab9b4a47f8d23f03"
server_key_version: "encryptionkey@0"
I20260928 22:53:25.696646 6410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "24fa558b3a1c425bb75f87e592beec33"
format_stamp: "Formatted at 2026-09-28 22:53:25 on dist-test-slave-56vw"
server_key: "2d957263cf6fe2d0421cb40cab3a7595"
server_key_iv: "eca874c74ab4f159ab9b4a47f8d23f03"
server_key_version: "encryptionkey@0"
I20260928 22:53:25.698107 6410 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:53:25.698900 6425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:25.699119 6410 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:53:25.699220 6410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "24fa558b3a1c425bb75f87e592beec33"
format_stamp: "Formatted at 2026-09-28 22:53:25 on dist-test-slave-56vw"
server_key: "2d957263cf6fe2d0421cb40cab3a7595"
server_key_iv: "eca874c74ab4f159ab9b4a47f8d23f03"
server_key_version: "encryptionkey@0"
I20260928 22:53:25.699311 6410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:25.709661 6410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:25.712287 6410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:25.712489 6410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:25.716835 6410 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.190:42455
I20260928 22:53:25.716867 6477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.190:42455 every 8 connection(s)
I20260928 22:53:25.717252 6410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:53:25.717530 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 6410
I20260928 22:53:25.717661 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:53:25.717873 16870 external_mini_cluster.cc:1428] Setting key 07bf5849e545c8fa68369e2681105fbf
I20260928 22:53:25.718340 6478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:25.721858 6478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24fa558b3a1c425bb75f87e592beec33: Bootstrap starting.
I20260928 22:53:25.722494 6478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24fa558b3a1c425bb75f87e592beec33: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:25.722815 6478 log.cc:824] T 00000000000000000000000000000000 P 24fa558b3a1c425bb75f87e592beec33: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:25.723695 6478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24fa558b3a1c425bb75f87e592beec33: No bootstrap required, opened a new log
I20260928 22:53:25.724110 6481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:25.718441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59905 (local address 127.16.121.190:42455)
0928 22:53:25.718658 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:25.718667 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:25.718701 (+ 34us) server_negotiation.cc:407] Connection header received
0928 22:53:25.719308 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:25.719318 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:25.719387 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:25.719549 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:25.719777 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:25.721169 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:25.722030 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:25.722455 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:25.722581 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:25.723388 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:25.723393 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:25.723393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:53:25.723564 (+ 171us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:25.723596 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:53:25.723607 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:25.723659 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:25.723801 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 22:53:25.723915 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":98,"threads_started":1}
I20260928 22:53:25.725437 16870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:53:25.725473 6478 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24fa558b3a1c425bb75f87e592beec33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24fa558b3a1c425bb75f87e592beec33" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 42455 } }
I20260928 22:53:25.725662 6478 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24fa558b3a1c425bb75f87e592beec33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:25.725713 6478 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24fa558b3a1c425bb75f87e592beec33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24fa558b3a1c425bb75f87e592beec33, State: Initialized, Role: FOLLOWER
I20260928 22:53:25.725836 6478 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24fa558b3a1c425bb75f87e592beec33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24fa558b3a1c425bb75f87e592beec33" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 42455 } }
I20260928 22:53:25.725946 6478 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24fa558b3a1c425bb75f87e592beec33 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:53:25.725994 6478 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24fa558b3a1c425bb75f87e592beec33 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:53:25.726045 6478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24fa558b3a1c425bb75f87e592beec33 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:25.726636 6478 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24fa558b3a1c425bb75f87e592beec33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24fa558b3a1c425bb75f87e592beec33" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 42455 } }
I20260928 22:53:25.726770 6478 leader_election.cc:302] T 00000000000000000000000000000000 P 24fa558b3a1c425bb75f87e592beec33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24fa558b3a1c425bb75f87e592beec33; no voters:
I20260928 22:53:25.726941 6478 leader_election.cc:288] T 00000000000000000000000000000000 P 24fa558b3a1c425bb75f87e592beec33 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:53:25.726997 6487 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24fa558b3a1c425bb75f87e592beec33 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:53:25.727267 6487 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24fa558b3a1c425bb75f87e592beec33 [term 1 LEADER]: Becoming Leader. State: Replica: 24fa558b3a1c425bb75f87e592beec33, State: Running, Role: LEADER
I20260928 22:53:25.727309 6478 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24fa558b3a1c425bb75f87e592beec33 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:53:25.727375 6487 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24fa558b3a1c425bb75f87e592beec33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24fa558b3a1c425bb75f87e592beec33" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 42455 } }
I20260928 22:53:25.727738 6487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24fa558b3a1c425bb75f87e592beec33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24fa558b3a1c425bb75f87e592beec33. Latest consensus state: current_term: 1 leader_uuid: "24fa558b3a1c425bb75f87e592beec33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24fa558b3a1c425bb75f87e592beec33" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 42455 } } }
I20260928 22:53:25.727813 6487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24fa558b3a1c425bb75f87e592beec33 [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:25.727954 6489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24fa558b3a1c425bb75f87e592beec33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24fa558b3a1c425bb75f87e592beec33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24fa558b3a1c425bb75f87e592beec33" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 42455 } } }
I20260928 22:53:25.728006 6489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24fa558b3a1c425bb75f87e592beec33 [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:25.728103 6495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:53:25.728621 6495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:53:25.730544 6481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:25.727089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38261 (local address 127.16.121.190:42455)
0928 22:53:25.727131 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:25.727133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:25.727199 (+ 66us) server_negotiation.cc:407] Connection header received
0928 22:53:25.727300 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:25.727301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:25.727312 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:25.727343 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:25.727558 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:25.728282 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:25.729418 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:25.729615 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:25.729669 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:25.730323 (+ 654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:25.730324 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:25.730325 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:53:25.730328 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:25.730342 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:53:25.730345 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:25.730390 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:25.730472 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 22:53:25.730494 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:53:25.732610 6495 catalog_manager.cc:1347] Generated new cluster ID: f4122894e6f94a40b3a57947573a03ec
I20260928 22:53:25.732668 6495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:53:25.754451 6495 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:53:25.755116 6495 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:53:25.759068 6495 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24fa558b3a1c425bb75f87e592beec33: Generated new TSK 0
I20260928 22:53:25.759387 6495 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:53:25.762014 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 6410
2026-09-28T22:53:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1350 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635733329745-16870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:53:25 dist-test-slave-56vw krb5kdc[6509](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:53:25 dist-test-slave-56vw krb5kdc[6509](info): set up 2 sockets
Sep 28 22:53:25 dist-test-slave-56vw krb5kdc[6509](info): commencing operation
krb5kdc: starting...
W20260928 22:53:26.815385 16870 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Sep 28 22:53:26 dist-test-slave-56vw krb5kdc[6509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:53:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:53:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.121.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
I20260928 22:53:26.947283 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:33327
--webserver_interface=127.16.121.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:38067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:33327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:27.030980 6522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:27.031255 6522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:27.031288 6522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:27.032620 6522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:53:27.032668 6522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:27.032683 6522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:27.032696 6522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:53:27.032709 6522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:53:27.034266 6522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:38067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:33327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:33327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.121.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:27.034494 6522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:27.034687 6522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:27.039829 6530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:27.039829 6528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:27.039852 6527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:27.040158 6522 server_base.cc:1034] running on GCE node
I20260928 22:53:27.040490 6522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:27.040834 6522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:27.042019 6522 hybrid_clock.cc:648] HybridClock initialized: now 1790636007041983 us; error 54 us; skew 500 ppm
Sep 28 22:53:27 dist-test-slave-56vw krb5kdc[6509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636007, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:27.052807 6522 init.cc:377] Logged in from keytab as kudu/127.16.121.190@KRBTEST.COM (short username kudu)
I20260928 22:53:27.053401 6522 webserver.cc:460] Webserver started at http://127.16.121.190:34103/ using document root <none> and password file <none>
I20260928 22:53:27.053659 6522 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:27.053730 6522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:27.053910 6522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:27.055133 6522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "7a506772c4894149b940d5427d6f3e2a"
format_stamp: "Formatted at 2026-09-28 22:53:27 on dist-test-slave-56vw"
server_key: "11da2869185e6f7c8cd96259bff95bb3"
server_key_iv: "b374597e138a8fde733746b23bd0ab42"
server_key_version: "encryptionkey@0"
I20260928 22:53:27.055508 6522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "7a506772c4894149b940d5427d6f3e2a"
format_stamp: "Formatted at 2026-09-28 22:53:27 on dist-test-slave-56vw"
server_key: "11da2869185e6f7c8cd96259bff95bb3"
server_key_iv: "b374597e138a8fde733746b23bd0ab42"
server_key_version: "encryptionkey@0"
I20260928 22:53:27.056949 6522 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 22:53:27.057749 6537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:27.057987 6522 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:53:27.058117 6522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "7a506772c4894149b940d5427d6f3e2a"
format_stamp: "Formatted at 2026-09-28 22:53:27 on dist-test-slave-56vw"
server_key: "11da2869185e6f7c8cd96259bff95bb3"
server_key_iv: "b374597e138a8fde733746b23bd0ab42"
server_key_version: "encryptionkey@0"
I20260928 22:53:27.058187 6522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:27.062152 6522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:27.064519 6522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:27.064692 6522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:27.069732 6522 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.190:33327
I20260928 22:53:27.069793 6589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.190:33327 every 8 connection(s)
I20260928 22:53:27.070199 6522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:53:27.071293 6590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:27.072649 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 6522
I20260928 22:53:27.072772 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:53:27.072978 16870 external_mini_cluster.cc:1428] Setting key 3bf0024332744556a6f3487395d37199
I20260928 22:53:27.074083 6590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a506772c4894149b940d5427d6f3e2a: Bootstrap starting.
I20260928 22:53:27.074712 6590 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a506772c4894149b940d5427d6f3e2a: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:27.075084 6590 log.cc:824] T 00000000000000000000000000000000 P 7a506772c4894149b940d5427d6f3e2a: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:27.075693 6593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:27.073541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42743 (local address 127.16.121.190:33327)
0928 22:53:27.073768 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:27.073777 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:27.073803 (+ 26us) server_negotiation.cc:407] Connection header received
0928 22:53:27.074272 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:27.074282 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:27.074318 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:27.074461 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:27.075259 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:27.075265 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:27.075266 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:53:27.075279 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:27.075325 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:53:27.075340 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:27.075444 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:27.075488 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 22:53:27.075530 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":82,"threads_started":1}
I20260928 22:53:27.075989 6590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a506772c4894149b940d5427d6f3e2a: No bootstrap required, opened a new log
I20260928 22:53:27.076632 16870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:53:27.078692 6590 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a506772c4894149b940d5427d6f3e2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a506772c4894149b940d5427d6f3e2a" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 33327 } }
I20260928 22:53:27.078893 6590 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a506772c4894149b940d5427d6f3e2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 22:53:27.078936 6590 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a506772c4894149b940d5427d6f3e2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a506772c4894149b940d5427d6f3e2a, State: Initialized, Role: FOLLOWER
I20260928 22:53:27.079099 6590 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a506772c4894149b940d5427d6f3e2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a506772c4894149b940d5427d6f3e2a" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 33327 } }
I20260928 22:53:27.079206 6590 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a506772c4894149b940d5427d6f3e2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 22:53:27.079241 6590 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a506772c4894149b940d5427d6f3e2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 22:53:27.079294 6590 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a506772c4894149b940d5427d6f3e2a [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:27.079599 6593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:27.078413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39655 (local address 127.16.121.190:33327)
0928 22:53:27.078466 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:27.078467 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:27.078602 (+ 135us) server_negotiation.cc:407] Connection header received
0928 22:53:27.078662 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:27.078663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:27.078668 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:27.078716 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:27.079390 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:27.079392 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:27.079392 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 22:53:27.079397 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:27.079412 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 22:53:27.079416 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:27.079463 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:27.079546 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 22:53:27.079557 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 22:53:27.080072 6590 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a506772c4894149b940d5427d6f3e2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a506772c4894149b940d5427d6f3e2a" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 33327 } }
I20260928 22:53:27.080185 6590 leader_election.cc:302] T 00000000000000000000000000000000 P 7a506772c4894149b940d5427d6f3e2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a506772c4894149b940d5427d6f3e2a; no voters:
I20260928 22:53:27.080344 6590 leader_election.cc:288] T 00000000000000000000000000000000 P 7a506772c4894149b940d5427d6f3e2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 22:53:27.080363 6600 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a506772c4894149b940d5427d6f3e2a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:53:27.080492 6600 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a506772c4894149b940d5427d6f3e2a [term 1 LEADER]: Becoming Leader. State: Replica: 7a506772c4894149b940d5427d6f3e2a, State: Running, Role: LEADER
I20260928 22:53:27.080581 6600 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a506772c4894149b940d5427d6f3e2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a506772c4894149b940d5427d6f3e2a" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 33327 } }
I20260928 22:53:27.080610 6590 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a506772c4894149b940d5427d6f3e2a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:53:27.080982 6601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a506772c4894149b940d5427d6f3e2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a506772c4894149b940d5427d6f3e2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a506772c4894149b940d5427d6f3e2a" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 33327 } } }
I20260928 22:53:27.081090 6601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a506772c4894149b940d5427d6f3e2a [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:27.080978 6602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a506772c4894149b940d5427d6f3e2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a506772c4894149b940d5427d6f3e2a. Latest consensus state: current_term: 1 leader_uuid: "7a506772c4894149b940d5427d6f3e2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a506772c4894149b940d5427d6f3e2a" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 33327 } } }
I20260928 22:53:27.081254 6602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a506772c4894149b940d5427d6f3e2a [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:27.081379 6606 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:53:27.082082 6606 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:53:27.083524 6606 catalog_manager.cc:1347] Generated new cluster ID: 19154f55739247caa774571fddf39690
I20260928 22:53:27.083616 6606 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:53:27.090025 6606 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:53:27.090193 6606 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:53:27.096685 6606 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a506772c4894149b940d5427d6f3e2a: Generated new TSK 0
I20260928 22:53:27.096930 6606 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 22:53:27.101995 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 6522
2026-09-28T22:53:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1340 ms)
[----------] 18 tests from AuthTokenIssuingTest (24561 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:53:27 dist-test-slave-56vw krb5kdc[6621](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:53:27 dist-test-slave-56vw krb5kdc[6621](info): set up 2 sockets
Sep 28 22:53:27 dist-test-slave-56vw krb5kdc[6621](info): commencing operation
krb5kdc: starting...
W20260928 22:53:28.167397 16870 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Sep 28 22:53:28 dist-test-slave-56vw krb5kdc[6621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:53:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:53:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.121.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
I20260928 22:53:28.294541 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:35875
--webserver_interface=127.16.121.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:40931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:35875,127.16.121.189:33713,127.16.121.188:39123
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:28.376765 6634 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 22:53:28.376989 6634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:28.377007 6634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:28.377027 6634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:28.378242 6634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:53:28.378288 6634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:28.378301 6634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:28.378314 6634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:53:28.378326 6634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:53:28.379916 6634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:40931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:35875,127.16.121.189:33713,127.16.121.188:39123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:35875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.121.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:28.380185 6634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:28.380411 6634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:28.385550 6640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:28.385563 6642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:28.385557 6639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:28.385879 6634 server_base.cc:1034] running on GCE node
I20260928 22:53:28.386087 6634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:28.386348 6634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:28.387486 6634 hybrid_clock.cc:648] HybridClock initialized: now 1790636008387464 us; error 41 us; skew 500 ppm
Sep 28 22:53:28 dist-test-slave-56vw krb5kdc[6621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636008, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:28.390152 6634 init.cc:377] Logged in from keytab as kudu/127.16.121.190@KRBTEST.COM (short username kudu)
I20260928 22:53:28.390595 6634 webserver.cc:460] Webserver started at http://127.16.121.190:40637/ using document root <none> and password file <none>
I20260928 22:53:28.390841 6634 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:28.390887 6634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:28.391075 6634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:28.392247 6634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "05ee4f38e4374d0faf7b64b8fddf9fcf"
format_stamp: "Formatted at 2026-09-28 22:53:28 on dist-test-slave-56vw"
server_key: "036a1f4ab675db5e4442a7c4d3fed7ed"
server_key_iv: "90182b035a008e07500359bb2b575b75"
server_key_version: "encryptionkey@0"
I20260928 22:53:28.392614 6634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "05ee4f38e4374d0faf7b64b8fddf9fcf"
format_stamp: "Formatted at 2026-09-28 22:53:28 on dist-test-slave-56vw"
server_key: "036a1f4ab675db5e4442a7c4d3fed7ed"
server_key_iv: "90182b035a008e07500359bb2b575b75"
server_key_version: "encryptionkey@0"
I20260928 22:53:28.393994 6634 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:53:28.394884 6649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:28.395150 6634 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:53:28.395256 6634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "05ee4f38e4374d0faf7b64b8fddf9fcf"
format_stamp: "Formatted at 2026-09-28 22:53:28 on dist-test-slave-56vw"
server_key: "036a1f4ab675db5e4442a7c4d3fed7ed"
server_key_iv: "90182b035a008e07500359bb2b575b75"
server_key_version: "encryptionkey@0"
I20260928 22:53:28.395344 6634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:28.405004 6634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:28.407511 6634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:28.407688 6634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:28.411755 6634 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.190:35875
I20260928 22:53:28.411792 6701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.190:35875 every 8 connection(s)
I20260928 22:53:28.412156 6634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:53:28.412971 6702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:28.414717 6702 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.121.190" port: 35875 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:53:28.419543 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 6634
I20260928 22:53:28.419663 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:53:28.419878 16870 external_mini_cluster.cc:1428] Setting key 294035609c5ff1746e688deef9d4fdc7
Sep 28 22:53:28 dist-test-slave-56vw krb5kdc[6621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636008, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:53:28.425644 6704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:28.415569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:35875 (local address 127.0.0.1:34382)
0928 22:53:28.415738 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:28.415782 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:28.416260 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:28.416550 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:28.416552 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:28.416819 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:28.416984 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:28.416988 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:28.418045 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:28.418046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:28.418768 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:28.418770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:28.419448 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:28.419450 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:28.419536 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:28.420149 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:28.420160 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:28.422013 (+ 1853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:28.424142 (+ 2129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:28.424148 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:28.424151 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:28.424356 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:28.424599 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:28.424600 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:28.424600 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:28.424691 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:28.425046 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:28.425050 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:28.425113 (+ 63us) client_negotiation.cc:769] Sending connection context
0928 22:53:28.425226 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 22:53:28.425454 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":82,"thread_start_us":89,"threads_started":1}
I20260928 22:53:28.425642 6705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:28.415687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34382 (local address 127.16.121.190:35875)
0928 22:53:28.415841 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:28.415846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:28.415886 (+ 40us) server_negotiation.cc:407] Connection header received
0928 22:53:28.416413 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:28.416415 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:28.416433 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:28.416526 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:28.417066 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:28.417945 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:28.418859 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:28.419356 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:28.419444 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:28.422111 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:28.422119 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:28.422120 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:28.422128 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:28.424045 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:28.424448 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:28.424451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:28.424452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:28.424533 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:28.424753 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:28.424754 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:28.424754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:28.424978 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:28.425068 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:28.425256 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 22:53:28.425391 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":95,"threads_started":1}
I20260928 22:53:28.426411 6702 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.121.189" port: 33713 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:53:28.426781 6704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:28.426664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.189:33713 (local address 127.0.0.1:58322)
0928 22:53:28.426713 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:28.426757 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.189:33713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 22:53:28.426937 6651 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.121.189:33713: connect: Connection refused (error 111)
I20260928 22:53:28.427446 6704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:28.427380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.189:33713 (local address 127.0.0.1:58332)
0928 22:53:28.427406 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:28.427429 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.189:33713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260928 22:53:28.427553 6702 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.121.189:33713: Network error: Client connection negotiation failed: client connection to 127.16.121.189:33713: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.16.121.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.189.keytab.
Entry for principal kudu/127.16.121.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.189.keytab.
I20260928 22:53:28.446969 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.189:33713
--webserver_interface=127.16.121.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.189.keytab
--principal=kudu/127.16.121.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:40931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:35875,127.16.121.189:33713,127.16.121.188:39123
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:53:28.476784 6702 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.121.189" port: 33713 } attempt: 1
I20260928 22:53:28.477418 6704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:28.477273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.189:33713 (local address 127.0.0.1:58336)
0928 22:53:28.477345 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:28.477383 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.189:33713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 22:53:28.477763 6704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:28.477657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.189:33713 (local address 127.0.0.1:58350)
0928 22:53:28.477713 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:28.477743 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.189:33713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 22:53:28.477855 6702 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.121.189:33713: Network error: Client connection negotiation failed: client connection to 127.16.121.189:33713: connect: Connection refused (error 111)
W20260928 22:53:28.529481 6709 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 22:53:28.529750 6709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:28.529781 6709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:28.529820 6709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:28.531001 6709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:53:28.531097 6709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:28.531134 6709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:28.531158 6709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:53:28.531195 6709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:53:28.532671 6709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:40931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:35875,127.16.121.189:33713,127.16.121.188:39123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.189.keytab
--principal=kudu/127.16.121.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.189:33713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.121.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:28.532941 6709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:28.533162 6709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:28.538424 6715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:28.538496 6717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:28.538434 6714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:28.538707 6709 server_base.cc:1034] running on GCE node
I20260928 22:53:28.538956 6709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:28.539343 6709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:28.540504 6709 hybrid_clock.cc:648] HybridClock initialized: now 1790636008540496 us; error 30 us; skew 500 ppm
Sep 28 22:53:28 dist-test-slave-56vw krb5kdc[6621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636008, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:28.543187 6709 init.cc:377] Logged in from keytab as kudu/127.16.121.189@KRBTEST.COM (short username kudu)
I20260928 22:53:28.543751 6709 webserver.cc:460] Webserver started at http://127.16.121.189:34863/ using document root <none> and password file <none>
I20260928 22:53:28.543992 6709 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:28.544055 6709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:28.544210 6709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:28.545394 6709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-1/data/instance:
uuid: "c62570af9dd641179a669ff889e48930"
format_stamp: "Formatted at 2026-09-28 22:53:28 on dist-test-slave-56vw"
server_key: "e09a5bf2066c96c3e348c619464d6240"
server_key_iv: "d6b72dd6a5eace06b2fc01c4806b8244"
server_key_version: "encryptionkey@0"
I20260928 22:53:28.545769 6709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-1/wal/instance:
uuid: "c62570af9dd641179a669ff889e48930"
format_stamp: "Formatted at 2026-09-28 22:53:28 on dist-test-slave-56vw"
server_key: "e09a5bf2066c96c3e348c619464d6240"
server_key_iv: "d6b72dd6a5eace06b2fc01c4806b8244"
server_key_version: "encryptionkey@0"
I20260928 22:53:28.546003 6702 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.121.189" port: 33713 } attempt: 2
I20260928 22:53:28.546454 6704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:28.546322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.189:33713 (local address 127.0.0.1:58352)
0928 22:53:28.546385 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:28.546419 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.189:33713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 22:53:28.546778 6704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:28.546674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.189:33713 (local address 127.0.0.1:58368)
0928 22:53:28.546715 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:28.546754 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.189:33713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 22:53:28.546865 6702 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.121.189:33713: Network error: Client connection negotiation failed: client connection to 127.16.121.189:33713: connect: Connection refused (error 111)
I20260928 22:53:28.547406 6709 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:53:28.548141 6724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:28.548365 6709 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:53:28.548455 6709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-1/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-1/wal
uuid: "c62570af9dd641179a669ff889e48930"
format_stamp: "Formatted at 2026-09-28 22:53:28 on dist-test-slave-56vw"
server_key: "e09a5bf2066c96c3e348c619464d6240"
server_key_iv: "d6b72dd6a5eace06b2fc01c4806b8244"
server_key_version: "encryptionkey@0"
I20260928 22:53:28.548532 6709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:28.566402 6709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:28.568612 6709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:28.568769 6709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:28.572999 6709 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.189:33713
I20260928 22:53:28.573035 6776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.189:33713 every 8 connection(s)
I20260928 22:53:28.573401 6709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-1/data/info.pb
I20260928 22:53:28.574342 6777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:28.575961 6777 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.121.190" port: 35875 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 22:53:28 dist-test-slave-56vw krb5kdc[6621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636008, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.189@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:53:28.582226 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 6709
I20260928 22:53:28.582321 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-1/wal/instance
I20260928 22:53:28.582525 16870 external_mini_cluster.cc:1428] Setting key cab071d82c46bce9c962ec336c67486a
I20260928 22:53:28.586298 6705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:28.576762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34396 (local address 127.16.121.190:35875)
0928 22:53:28.576833 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:28.576835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:28.577036 (+ 201us) server_negotiation.cc:407] Connection header received
0928 22:53:28.577510 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:28.577511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:28.577522 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:28.577591 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:28.578194 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:28.578723 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:28.579851 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:28.580051 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:28.580111 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:28.582829 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:28.582830 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:28.582831 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:28.582839 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:28.584513 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:28.585104 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:28.585105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:28.585105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:28.585185 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:28.585530 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:28.585532 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:28.585532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:28.585663 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:28.585757 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:28.586122 (+ 365us) server_negotiation.cc:299] Negotiation successful
0928 22:53:28.586222 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 22:53:28.586598 6779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:28.576720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:35875 (local address 127.0.0.1:34396)
0928 22:53:28.576894 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:28.576939 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:28.577427 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:28.577647 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:28.577650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:28.577910 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:28.578110 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:28.578114 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:28.578824 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:28.578825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:28.579763 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:28.579766 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:28.580145 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:28.580146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:28.580266 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:28.580831 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:28.580839 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:28.582726 (+ 1887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:28.584610 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:28.584621 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:28.584624 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:28.584989 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:28.585267 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:28.585268 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:28.585268 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:28.585428 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:28.585756 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:28.585764 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:28.585952 (+ 188us) client_negotiation.cc:769] Sending connection context
0928 22:53:28.586076 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 22:53:28.586298 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":96,"threads_started":1}
I20260928 22:53:28.587386 6777 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.121.189" port: 33713 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.121.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.188@KRBTEST.COM" created.
Sep 28 22:53:28 dist-test-slave-56vw krb5kdc[6621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636008, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.189@KRBTEST.COM for kudu/127.16.121.189@KRBTEST.COM
I20260928 22:53:28.598131 6779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:28.587681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.189:33713 (local address 127.0.0.1:58370)
0928 22:53:28.587949 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:28.587963 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:28.588012 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:28.588175 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:28.588175 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:28.588371 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:28.588447 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:28.588449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:28.589394 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:28.589395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:28.590265 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:28.590268 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:28.590803 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:28.590804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:28.590912 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:28.591495 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:28.591501 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:28.594937 (+ 3436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:28.596988 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:28.596990 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:28.596990 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:28.597162 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:28.597387 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:28.597387 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:28.597388 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:28.597443 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:28.597748 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:28.597749 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:28.597789 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 22:53:28.597863 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 22:53:28.597972 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249}
I20260928 22:53:28.598147 6781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:28.587773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58370 (local address 127.16.121.189:33713)
0928 22:53:28.587992 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:28.587996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:28.588017 (+ 21us) server_negotiation.cc:407] Connection header received
0928 22:53:28.588078 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:28.588079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:28.588092 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:28.588157 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:28.588543 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:28.589301 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:28.590364 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:28.590719 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:28.590785 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:28.595086 (+ 4301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:28.595103 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:28.595103 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:28.595112 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:28.596871 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:28.597248 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:28.597252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:28.597252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:28.597333 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:28.597501 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:28.597502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:28.597503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:28.597675 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:28.597774 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:28.597895 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 22:53:28.598042 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":175,"thread_start_us":110,"threads_started":1}
I20260928 22:53:28.598896 6777 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.121.188" port: 39123 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:53:28.599375 6779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:28.599242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:39123 (local address 127.0.0.1:44062)
0928 22:53:28.599293 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:28.599345 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.188:39123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 22:53:28.599509 6727 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.121.188:39123: connect: Connection refused (error 111)
I20260928 22:53:28.603463 6779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:28.603309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:39123 (local address 127.0.0.1:44074)
0928 22:53:28.603380 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:28.603412 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.188:39123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 22:53:28.603631 6777 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.121.188:39123: Network error: Client connection negotiation failed: client connection to 127.16.121.188:39123: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.188.keytab.
Entry for principal kudu/127.16.121.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.188.keytab.
I20260928 22:53:28.624219 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.188:39123
--webserver_interface=127.16.121.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.188.keytab
--principal=kudu/127.16.121.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:40931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:35875,127.16.121.189:33713,127.16.121.188:39123
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:53:28.638006 6702 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.121.189" port: 33713 } attempt: 3
Sep 28 22:53:28 dist-test-slave-56vw krb5kdc[6621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636008, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for kudu/127.16.121.189@KRBTEST.COM
I20260928 22:53:28.645690 6704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:28.638473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.189:33713 (local address 127.0.0.1:58382)
0928 22:53:28.638534 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:28.638545 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:28.638587 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:28.638834 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:28.638835 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:28.639009 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:28.639127 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:28.639129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:28.640004 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:28.640005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:28.640549 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:28.640551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:28.640959 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:28.640959 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:28.641033 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:28.641519 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:28.641524 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:28.642818 (+ 1294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:28.644591 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:28.644593 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:28.644593 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:28.644783 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:28.645005 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:28.645006 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:28.645006 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:28.645065 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:28.645337 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:28.645338 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:28.645390 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 22:53:28.645474 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 22:53:28.645583 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 22:53:28.645689 6781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:28.638653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58382 (local address 127.16.121.189:33713)
0928 22:53:28.638713 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:28.638719 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:28.638726 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:53:28.638750 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:28.638751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:28.638761 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:28.638827 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:28.639222 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:28.639917 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:28.640635 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:28.640872 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:28.640957 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:28.642917 (+ 1960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:28.642919 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:28.642919 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:28.642928 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:28.644501 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:28.644875 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:28.644876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:28.644877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:28.644934 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:28.645135 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:28.645136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:28.645137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:28.645268 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:28.645347 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:28.645500 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 22:53:28.645594 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 22:53:28.646546 6702 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.121.188" port: 39123 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:53:28.646946 6704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:28.646827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:39123 (local address 127.0.0.1:44080)
0928 22:53:28.646880 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:28.646912 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.188:39123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 22:53:28.647607 6704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:28.647500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:39123 (local address 127.0.0.1:44092)
0928 22:53:28.647545 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:28.647574 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.188:39123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 22:53:28.647782 6702 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.121.188:39123: Network error: Client connection negotiation failed: client connection to 127.16.121.188:39123: connect: Connection refused (error 111)
I20260928 22:53:28.653148 6777 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.121.188" port: 39123 } attempt: 1
I20260928 22:53:28.653618 6779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:28.653483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:39123 (local address 127.0.0.1:44094)
0928 22:53:28.653546 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:28.653577 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.188:39123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 22:53:28.654070 6779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:28.653976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:39123 (local address 127.0.0.1:44096)
0928 22:53:28.654015 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:28.654039 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.188:39123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 22:53:28.654513 6777 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.121.188:39123: Network error: Client connection negotiation failed: client connection to 127.16.121.188:39123: connect: Connection refused (error 111)
I20260928 22:53:28.678941 6702 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.121.188" port: 39123 } attempt: 1
I20260928 22:53:28.679464 6704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:28.679328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:39123 (local address 127.0.0.1:44108)
0928 22:53:28.679391 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:28.679431 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.188:39123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 22:53:28.679873 6704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:28.679794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:39123 (local address 127.0.0.1:44124)
0928 22:53:28.679832 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:28.679857 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.188:39123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 22:53:28.679973 6702 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.121.188:39123: Network error: Client connection negotiation failed: client connection to 127.16.121.188:39123: connect: Connection refused (error 111)
I20260928 22:53:28.722680 6777 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.121.188" port: 39123 } attempt: 2
I20260928 22:53:28.723284 6779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:28.723003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:39123 (local address 127.0.0.1:44140)
0928 22:53:28.723098 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:28.723243 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.188:39123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 22:53:28.723663 6779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:28.723561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:39123 (local address 127.0.0.1:44142)
0928 22:53:28.723598 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:28.723628 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.188:39123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260928 22:53:28.723773 6777 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.121.188:39123: Network error: Client connection negotiation failed: client connection to 127.16.121.188:39123: connect: Connection refused (error 111)
W20260928 22:53:28.728310 6784 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 22:53:28.728562 6784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:28.728621 6784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:28.728667 6784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:28.729889 6784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:53:28.729950 6784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:28.729976 6784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:28.730005 6784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:53:28.730031 6784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:53:28.731604 6784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:40931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:35875,127.16.121.189:33713,127.16.121.188:39123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.188.keytab
--principal=kudu/127.16.121.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.188:39123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.121.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:28.731878 6784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:28.732106 6784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 22:53:28.737128 6784 server_base.cc:1034] running on GCE node
W20260928 22:53:28.737113 6790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:28.737272 6789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:28.737105 6792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:28.737661 6784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:28.737951 6784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:28.739121 6784 hybrid_clock.cc:648] HybridClock initialized: now 1790636008739102 us; error 27 us; skew 500 ppm
Sep 28 22:53:28 dist-test-slave-56vw krb5kdc[6621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636008, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:28.741755 6784 init.cc:377] Logged in from keytab as kudu/127.16.121.188@KRBTEST.COM (short username kudu)
I20260928 22:53:28.742265 6784 webserver.cc:460] Webserver started at http://127.16.121.188:34961/ using document root <none> and password file <none>
I20260928 22:53:28.742511 6784 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:28.742579 6784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:28.742731 6784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:28.743903 6784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-2/data/instance:
uuid: "e90dc913ca484b06972a24d139392dd9"
format_stamp: "Formatted at 2026-09-28 22:53:28 on dist-test-slave-56vw"
server_key: "097fae772805539fd318491777bbf273"
server_key_iv: "417849ace6c04e0b7cd1f87fcebd7aae"
server_key_version: "encryptionkey@0"
I20260928 22:53:28.744287 6784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-2/wal/instance:
uuid: "e90dc913ca484b06972a24d139392dd9"
format_stamp: "Formatted at 2026-09-28 22:53:28 on dist-test-slave-56vw"
server_key: "097fae772805539fd318491777bbf273"
server_key_iv: "417849ace6c04e0b7cd1f87fcebd7aae"
server_key_version: "encryptionkey@0"
I20260928 22:53:28.745667 6784 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 22:53:28.746476 6799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:28.746657 6784 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 22:53:28.746747 6784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-2/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-2/wal
uuid: "e90dc913ca484b06972a24d139392dd9"
format_stamp: "Formatted at 2026-09-28 22:53:28 on dist-test-slave-56vw"
server_key: "097fae772805539fd318491777bbf273"
server_key_iv: "417849ace6c04e0b7cd1f87fcebd7aae"
server_key_version: "encryptionkey@0"
I20260928 22:53:28.746825 6784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:28.754042 6784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:28.755105 6702 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.121.188" port: 39123 } attempt: 2
I20260928 22:53:28.755501 6704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:28.755373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:39123 (local address 127.0.0.1:44152)
0928 22:53:28.755436 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:28.755471 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.188:39123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 22:53:28.755864 6704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:28.755723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:39123 (local address 127.0.0.1:44168)
0928 22:53:28.755789 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:28.755820 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.188:39123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 22:53:28.756016 6702 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.121.188:39123: Network error: Client connection negotiation failed: client connection to 127.16.121.188:39123: connect: Connection refused (error 111)
I20260928 22:53:28.756382 6784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:28.756524 6784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:28.760697 6784 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.188:39123
I20260928 22:53:28.760838 6851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.188:39123 every 8 connection(s)
I20260928 22:53:28.761107 6784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-2/data/info.pb
I20260928 22:53:28.761294 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 6784
I20260928 22:53:28.761402 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790635733329745-16870-0/minicluster-data/master-2/wal/instance
I20260928 22:53:28.761590 16870 external_mini_cluster.cc:1428] Setting key 2355845d022f79b5f932633d5d91d859
I20260928 22:53:28.762066 6852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:28.764223 6852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.121.190" port: 35875 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 22:53:28 dist-test-slave-56vw krb5kdc[6621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.121.188@KRBTEST.COM
Sep 28 22:53:28 dist-test-slave-56vw krb5kdc[6621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636008, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.188@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:53:28.773222 6855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:28.762114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44178 (local address 127.16.121.188:39123)
0928 22:53:28.762300 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:28.762309 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:28.762344 (+ 35us) server_negotiation.cc:407] Connection header received
0928 22:53:28.762782 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:28.762789 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:28.762838 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:28.763000 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:28.763659 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:28.764762 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:28.765943 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:28.766646 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:28.766768 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:28.769375 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:28.769381 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:28.769382 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:28.769388 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:28.771416 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:28.771821 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:28.771825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:28.771826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:28.771933 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:28.772258 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:28.772259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:28.772259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:28.772550 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:28.772679 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:28.772858 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 22:53:28.772999 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":85,"threads_started":1}
I20260928 22:53:28.773473 6705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:28.764931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34402 (local address 127.16.121.190:35875)
0928 22:53:28.764997 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:28.764999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:28.765238 (+ 239us) server_negotiation.cc:407] Connection header received
0928 22:53:28.765269 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:28.765270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:28.765280 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:28.765327 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:28.765952 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:28.766647 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:28.767810 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:28.768023 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:28.768092 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:28.770377 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:28.770378 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:28.770378 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:28.770386 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:28.771955 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:28.772456 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:28.772458 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:28.772458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:28.772515 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:28.772772 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:28.772772 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:28.772773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:28.772905 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:28.772978 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:28.773266 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 22:53:28.773353 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 22:53:28.773521 6856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:28.764844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:35875 (local address 127.0.0.1:34402)
0928 22:53:28.765078 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:28.765119 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:28.765187 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:28.765363 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:28.765367 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:28.765692 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:28.765780 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:28.765782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:28.766752 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:28.766753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:28.767679 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:28.767683 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:28.768113 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:28.768114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:28.768190 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:28.768735 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:28.768743 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:28.770288 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:28.772037 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:28.772045 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:28.772047 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:28.772381 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:28.772609 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:28.772610 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:28.772611 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:28.772687 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:28.773080 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:28.773088 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:28.773133 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 22:53:28.773210 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 22:53:28.773389 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":107,"threads_started":1}
I20260928 22:53:28.774055 6852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.121.189" port: 33713 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 22:53:28 dist-test-slave-56vw krb5kdc[6621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636008, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.188@KRBTEST.COM for kudu/127.16.121.189@KRBTEST.COM
Sep 28 22:53:28 dist-test-slave-56vw krb5kdc[6621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:53:28.781973 6856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:28.774312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.189:33713 (local address 127.0.0.1:58394)
0928 22:53:28.774820 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:28.774830 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:28.774864 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:28.774990 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:28.774991 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:28.775177 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:28.775226 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:28.775227 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:28.776050 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:28.776051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:28.776593 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:28.776595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:28.777010 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:28.777011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:28.777118 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:28.777505 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:28.777509 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:28.778715 (+ 1206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:28.780856 (+ 2141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:28.780857 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:28.780857 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:28.781027 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:28.781264 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:28.781265 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:28.781265 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:28.781341 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:28.781627 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:28.781628 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:28.781682 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 22:53:28.781759 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 22:53:28.781842 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494}
I20260928 22:53:28.782044 6781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:28.774377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58394 (local address 127.16.121.189:33713)
0928 22:53:28.774423 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:28.774425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:28.774881 (+ 456us) server_negotiation.cc:407] Connection header received
0928 22:53:28.774908 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:28.774909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:28.774920 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:28.774990 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:28.775305 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:28.775964 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:28.776682 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:28.776921 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:28.776996 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:28.778820 (+ 1824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:28.778822 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:28.778822 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:28.778829 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:28.780756 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:28.781109 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:28.781111 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:28.781111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:28.781177 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:28.781438 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:28.781438 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:28.781439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:28.781560 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:28.781618 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:28.781812 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 22:53:28.781905 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:53:28.782702 6705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:28.774182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34416 (local address 127.16.121.190:35875)
0928 22:53:28.774236 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:28.774238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:28.774247 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:53:28.774279 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:28.774279 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:28.774289 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:28.774354 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:28.774810 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:28.775615 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:28.776321 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:28.776518 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:28.776604 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:28.779401 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:28.779402 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:28.779403 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:28.779411 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:28.781333 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:28.781716 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:28.781717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:28.781717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:28.781804 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:28.782047 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:28.782048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:28.782049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:28.782212 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:28.782417 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:28.782562 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 22:53:28.782642 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 22:53:28.782845 6852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.121.188" port: 39123 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 22:53:28 dist-test-slave-56vw krb5kdc[6621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636008, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.188@KRBTEST.COM for kudu/127.16.121.188@KRBTEST.COM
Sep 28 22:53:28 dist-test-slave-56vw krb5kdc[6621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.121.189@KRBTEST.COM
I20260928 22:53:28.791445 6855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:28.783207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44194 (local address 127.16.121.188:39123)
0928 22:53:28.783242 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:28.783243 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:28.783250 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:53:28.783326 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:28.783327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:28.783336 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:28.783381 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:28.783626 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:28.784322 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:28.785237 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:28.785554 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:28.785635 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:28.787942 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:28.787944 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:28.787944 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:28.787952 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:28.789890 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:28.790294 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:28.790295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:28.790296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:28.790350 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:28.790574 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:28.790575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:28.790575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:28.790794 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:28.790885 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:28.791210 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 22:53:28.791330 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":22}
I20260928 22:53:28.791492 6856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:28.783155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:39123 (local address 127.0.0.1:44194)
0928 22:53:28.783186 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:28.783198 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:28.783270 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:28.783417 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:28.783418 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:28.783543 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:28.783572 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:28.783573 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:28.784397 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:28.784398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:28.785075 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:28.785077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:28.785631 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:28.785632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:28.785672 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:28.786043 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:28.786048 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:28.787841 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:28.789972 (+ 2131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:28.789973 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:28.789974 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:28.790166 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:28.790417 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:28.790418 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:28.790418 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:28.790486 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:28.790921 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:28.790923 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:28.790980 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 22:53:28.791120 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 22:53:28.791254 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15}
I20260928 22:53:28.792335 6781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:28.783387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58404 (local address 127.16.121.189:33713)
0928 22:53:28.783428 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:28.783431 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:28.783438 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:53:28.783495 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:28.783496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:28.783506 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:28.783560 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:28.784129 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:28.784818 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:28.785760 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:28.786057 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:28.786141 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:28.788899 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:28.788902 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:28.788902 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:28.788910 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:28.790444 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:28.790916 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:28.790918 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:28.790918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:28.790990 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:28.791357 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:28.791358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:28.791359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:28.791514 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:28.791972 (+ 458us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:28.792195 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 22:53:28.792274 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 22:53:28.793397 6852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e90dc913ca484b06972a24d139392dd9: Bootstrap starting.
I20260928 22:53:28.794216 16870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:53:28.794528 6852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e90dc913ca484b06972a24d139392dd9: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:28.794842 6852 log.cc:824] T 00000000000000000000000000000000 P e90dc913ca484b06972a24d139392dd9: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:28.795768 6852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e90dc913ca484b06972a24d139392dd9: No bootstrap required, opened a new log
I20260928 22:53:28.795586 6676 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "05ee4f38e4374d0faf7b64b8fddf9fcf"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34416
I20260928 22:53:28.798090 6852 raft_consensus.cc:348] T 00000000000000000000000000000000 P e90dc913ca484b06972a24d139392dd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ee4f38e4374d0faf7b64b8fddf9fcf" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 35875 } } peers { permanent_uuid: "c62570af9dd641179a669ff889e48930" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 33713 } } peers { permanent_uuid: "e90dc913ca484b06972a24d139392dd9" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 39123 } }
I20260928 22:53:28.798269 6852 raft_consensus.cc:729] T 00000000000000000000000000000000 P e90dc913ca484b06972a24d139392dd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e90dc913ca484b06972a24d139392dd9, State: Initialized, Role: FOLLOWER
I20260928 22:53:28.798354 6852 consensus_queue.cc:260] T 00000000000000000000000000000000 P e90dc913ca484b06972a24d139392dd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ee4f38e4374d0faf7b64b8fddf9fcf" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 35875 } } peers { permanent_uuid: "c62570af9dd641179a669ff889e48930" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 33713 } } peers { permanent_uuid: "e90dc913ca484b06972a24d139392dd9" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 39123 } }
I20260928 22:53:28.798715 6858 sys_catalog.cc:455] T 00000000000000000000000000000000 P e90dc913ca484b06972a24d139392dd9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ee4f38e4374d0faf7b64b8fddf9fcf" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 35875 } } peers { permanent_uuid: "c62570af9dd641179a669ff889e48930" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 33713 } } peers { permanent_uuid: "e90dc913ca484b06972a24d139392dd9" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 39123 } } }
I20260928 22:53:28.798764 6852 sys_catalog.cc:564] T 00000000000000000000000000000000 P e90dc913ca484b06972a24d139392dd9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:53:28.798802 6858 sys_catalog.cc:458] T 00000000000000000000000000000000 P e90dc913ca484b06972a24d139392dd9 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 22:53:28.800899 6869 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e90dc913ca484b06972a24d139392dd9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:53:28.800956 6869 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:53:28.801222 6784 master_runner.cc:186] Error getting master registration for 127.16.121.190:35875: OK, instance_id { permanent_uuid: "05ee4f38e4374d0faf7b64b8fddf9fcf" instance_seqno: 1790636008410540 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 22:53:28.804507 6784 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 22:53:28.814934 6777 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.121.188" port: 39123 } attempt: 3
Sep 28 22:53:28 dist-test-slave-56vw krb5kdc[6621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636008, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.189@KRBTEST.COM for kudu/127.16.121.188@KRBTEST.COM
I20260928 22:53:28.823653 6855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:28.815367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44200 (local address 127.16.121.188:39123)
0928 22:53:28.815441 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:28.815443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:28.815455 (+ 12us) server_negotiation.cc:407] Connection header received
0928 22:53:28.815542 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:28.815543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:28.815554 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:28.815631 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:28.816012 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:28.816742 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:28.817843 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:28.818147 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:28.818213 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:28.820749 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:28.820750 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:28.820751 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:28.820758 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:28.822468 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:28.822805 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:28.822807 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:28.822807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:28.822862 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:28.823081 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:28.823082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:28.823082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:28.823242 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:28.823322 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:28.823470 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 22:53:28.823551 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:53:28.823653 6779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:28.815326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:39123 (local address 127.0.0.1:44200)
0928 22:53:28.815398 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:28.815410 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:28.815457 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:28.815658 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:28.815659 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:28.815869 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:28.815927 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:28.815929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:28.816833 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:28.816834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:28.817749 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:28.817752 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:28.818228 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:28.818229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:28.818419 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:28.819017 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:28.819023 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:28.820657 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:28.822563 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:28.822564 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:28.822564 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:28.822733 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:28.822938 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:28.822939 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:28.822939 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:28.823000 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:28.823313 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:28.823314 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:28.823362 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 22:53:28.823447 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 22:53:28.823551 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 22:53:28.825315 6777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c62570af9dd641179a669ff889e48930: Bootstrap starting.
I20260928 22:53:28.826218 6777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c62570af9dd641179a669ff889e48930: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:28.826526 6777 log.cc:824] T 00000000000000000000000000000000 P c62570af9dd641179a669ff889e48930: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:28.827343 6777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c62570af9dd641179a669ff889e48930: No bootstrap required, opened a new log
I20260928 22:53:28.828807 6777 raft_consensus.cc:348] T 00000000000000000000000000000000 P c62570af9dd641179a669ff889e48930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ee4f38e4374d0faf7b64b8fddf9fcf" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 35875 } } peers { permanent_uuid: "c62570af9dd641179a669ff889e48930" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 33713 } } peers { permanent_uuid: "e90dc913ca484b06972a24d139392dd9" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 39123 } }
I20260928 22:53:28.828948 6777 raft_consensus.cc:729] T 00000000000000000000000000000000 P c62570af9dd641179a669ff889e48930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c62570af9dd641179a669ff889e48930, State: Initialized, Role: FOLLOWER
I20260928 22:53:28.829042 6777 consensus_queue.cc:260] T 00000000000000000000000000000000 P c62570af9dd641179a669ff889e48930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ee4f38e4374d0faf7b64b8fddf9fcf" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 35875 } } peers { permanent_uuid: "c62570af9dd641179a669ff889e48930" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 33713 } } peers { permanent_uuid: "e90dc913ca484b06972a24d139392dd9" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 39123 } }
I20260928 22:53:28.829399 6777 sys_catalog.cc:564] T 00000000000000000000000000000000 P c62570af9dd641179a669ff889e48930 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:53:28.829432 6872 sys_catalog.cc:455] T 00000000000000000000000000000000 P c62570af9dd641179a669ff889e48930 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ee4f38e4374d0faf7b64b8fddf9fcf" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 35875 } } peers { permanent_uuid: "c62570af9dd641179a669ff889e48930" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 33713 } } peers { permanent_uuid: "e90dc913ca484b06972a24d139392dd9" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 39123 } } }
I20260928 22:53:28.829545 6872 sys_catalog.cc:458] T 00000000000000000000000000000000 P c62570af9dd641179a669ff889e48930 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 22:53:28.831363 6883 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c62570af9dd641179a669ff889e48930: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:53:28.831439 6883 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:53:28.831481 6709 master_runner.cc:186] Error getting master registration for 127.16.121.190:35875: OK, instance_id { permanent_uuid: "05ee4f38e4374d0faf7b64b8fddf9fcf" instance_seqno: 1790636008410540 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 22:53:28.833887 6709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 22:53:28.855165 6702 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.121.188" port: 39123 } attempt: 3
Sep 28 22:53:28 dist-test-slave-56vw krb5kdc[6621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636008, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for kudu/127.16.121.188@KRBTEST.COM
I20260928 22:53:28.865635 6704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:28.855559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:39123 (local address 127.0.0.1:44214)
0928 22:53:28.855623 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:28.855635 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:28.855685 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:28.855837 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:28.855838 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:28.856015 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:28.856061 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:28.856063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:28.857276 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:28.857278 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:28.859893 (+ 2615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:28.859898 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:28.860305 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:28.860307 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:28.860368 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:28.860899 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:28.860905 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:28.862581 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:28.864567 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:28.864568 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:28.864568 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:28.864756 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:28.864986 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:28.864987 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:28.864987 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:28.865046 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:28.865295 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:28.865296 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:28.865348 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 22:53:28.865413 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 22:53:28.865522 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 22:53:28.866001 6855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:28.855610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44214 (local address 127.16.121.188:39123)
0928 22:53:28.855654 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:28.855656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:28.855669 (+ 13us) server_negotiation.cc:407] Connection header received
0928 22:53:28.855764 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:28.855765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:28.855776 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:28.855830 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:28.856142 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:28.857124 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:28.860010 (+ 2886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:28.860208 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:28.860305 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:28.862662 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:28.862663 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:28.862664 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:28.862671 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:28.864492 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:28.864848 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:28.864850 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:28.864850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:28.864909 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:28.865094 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:28.865095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:28.865095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:28.865220 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:28.865301 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:28.865837 (+ 536us) server_negotiation.cc:299] Negotiation successful
0928 22:53:28.865911 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 22:53:28.867868 6702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05ee4f38e4374d0faf7b64b8fddf9fcf: Bootstrap starting.
I20260928 22:53:28.868623 6702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05ee4f38e4374d0faf7b64b8fddf9fcf: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:28.868919 6702 log.cc:824] T 00000000000000000000000000000000 P 05ee4f38e4374d0faf7b64b8fddf9fcf: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:28.869725 6702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05ee4f38e4374d0faf7b64b8fddf9fcf: No bootstrap required, opened a new log
I20260928 22:53:28.870766 6702 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05ee4f38e4374d0faf7b64b8fddf9fcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ee4f38e4374d0faf7b64b8fddf9fcf" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 35875 } } peers { permanent_uuid: "c62570af9dd641179a669ff889e48930" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 33713 } } peers { permanent_uuid: "e90dc913ca484b06972a24d139392dd9" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 39123 } }
I20260928 22:53:28.870909 6702 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05ee4f38e4374d0faf7b64b8fddf9fcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05ee4f38e4374d0faf7b64b8fddf9fcf, State: Initialized, Role: FOLLOWER
I20260928 22:53:28.871003 6702 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05ee4f38e4374d0faf7b64b8fddf9fcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ee4f38e4374d0faf7b64b8fddf9fcf" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 35875 } } peers { permanent_uuid: "c62570af9dd641179a669ff889e48930" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 33713 } } peers { permanent_uuid: "e90dc913ca484b06972a24d139392dd9" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 39123 } }
I20260928 22:53:28.871356 6886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05ee4f38e4374d0faf7b64b8fddf9fcf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ee4f38e4374d0faf7b64b8fddf9fcf" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 35875 } } peers { permanent_uuid: "c62570af9dd641179a669ff889e48930" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 33713 } } peers { permanent_uuid: "e90dc913ca484b06972a24d139392dd9" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 39123 } } }
I20260928 22:53:28.871425 6702 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05ee4f38e4374d0faf7b64b8fddf9fcf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:53:28.871454 6886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05ee4f38e4374d0faf7b64b8fddf9fcf [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:53:28.872816 6676 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05ee4f38e4374d0faf7b64b8fddf9fcf [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 22:53:28.872887 6676 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05ee4f38e4374d0faf7b64b8fddf9fcf [term 0 FOLLOWER]: Advancing to term 1
W20260928 22:53:28.873193 6897 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 05ee4f38e4374d0faf7b64b8fddf9fcf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:53:28.873353 6897 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:53:28.873780 6676 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05ee4f38e4374d0faf7b64b8fddf9fcf [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ee4f38e4374d0faf7b64b8fddf9fcf" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 35875 } } peers { permanent_uuid: "c62570af9dd641179a669ff889e48930" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 33713 } } peers { permanent_uuid: "e90dc913ca484b06972a24d139392dd9" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 39123 } }
I20260928 22:53:28.874363 6676 leader_election.cc:288] T 00000000000000000000000000000000 P 05ee4f38e4374d0faf7b64b8fddf9fcf [CANDIDATE]: Term 1 election: Requested vote from peers c62570af9dd641179a669ff889e48930 (127.16.121.189:33713), e90dc913ca484b06972a24d139392dd9 (127.16.121.188:39123)
I20260928 22:53:28.874437 6826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "05ee4f38e4374d0faf7b64b8fddf9fcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e90dc913ca484b06972a24d139392dd9"
I20260928 22:53:28.874575 6826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e90dc913ca484b06972a24d139392dd9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:28.875689 6826 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e90dc913ca484b06972a24d139392dd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05ee4f38e4374d0faf7b64b8fddf9fcf in term 1.
I20260928 22:53:28.876200 6751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "05ee4f38e4374d0faf7b64b8fddf9fcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c62570af9dd641179a669ff889e48930"
I20260928 22:53:28.876354 6751 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c62570af9dd641179a669ff889e48930 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:28.876749 6652 leader_election.cc:302] T 00000000000000000000000000000000 P 05ee4f38e4374d0faf7b64b8fddf9fcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05ee4f38e4374d0faf7b64b8fddf9fcf, e90dc913ca484b06972a24d139392dd9; no voters:
I20260928 22:53:28.877328 6751 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c62570af9dd641179a669ff889e48930 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05ee4f38e4374d0faf7b64b8fddf9fcf in term 1.
I20260928 22:53:28.877468 6886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05ee4f38e4374d0faf7b64b8fddf9fcf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:53:28.877609 6886 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05ee4f38e4374d0faf7b64b8fddf9fcf [term 1 LEADER]: Becoming Leader. State: Replica: 05ee4f38e4374d0faf7b64b8fddf9fcf, State: Running, Role: LEADER
I20260928 22:53:28.877692 6886 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05ee4f38e4374d0faf7b64b8fddf9fcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ee4f38e4374d0faf7b64b8fddf9fcf" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 35875 } } peers { permanent_uuid: "c62570af9dd641179a669ff889e48930" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 33713 } } peers { permanent_uuid: "e90dc913ca484b06972a24d139392dd9" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 39123 } }
I20260928 22:53:28.878139 6886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05ee4f38e4374d0faf7b64b8fddf9fcf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05ee4f38e4374d0faf7b64b8fddf9fcf. Latest consensus state: current_term: 1 leader_uuid: "05ee4f38e4374d0faf7b64b8fddf9fcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ee4f38e4374d0faf7b64b8fddf9fcf" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 35875 } } peers { permanent_uuid: "c62570af9dd641179a669ff889e48930" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 33713 } } peers { permanent_uuid: "e90dc913ca484b06972a24d139392dd9" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 39123 } } }
I20260928 22:53:28.878206 6886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05ee4f38e4374d0faf7b64b8fddf9fcf [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:28.880404 6905 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:53:28.880596 6634 master_runner.cc:263] No leader master found from master 05ee4f38e4374d0faf7b64b8fddf9fcf
I20260928 22:53:28.880642 6634 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 22:53:28.880683 6905 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:53:28.883327 6826 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e90dc913ca484b06972a24d139392dd9 [term 1 FOLLOWER]: Refusing update from remote peer 05ee4f38e4374d0faf7b64b8fddf9fcf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:53:28.883514 6751 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c62570af9dd641179a669ff889e48930 [term 1 FOLLOWER]: Refusing update from remote peer 05ee4f38e4374d0faf7b64b8fddf9fcf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:53:28.883677 6886 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 05ee4f38e4374d0faf7b64b8fddf9fcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e90dc913ca484b06972a24d139392dd9" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 39123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:53:28.883944 6886 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 05ee4f38e4374d0faf7b64b8fddf9fcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c62570af9dd641179a669ff889e48930" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 33713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:53:28.885564 6872 sys_catalog.cc:455] T 00000000000000000000000000000000 P c62570af9dd641179a669ff889e48930 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05ee4f38e4374d0faf7b64b8fddf9fcf. Latest consensus state: current_term: 1 leader_uuid: "05ee4f38e4374d0faf7b64b8fddf9fcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ee4f38e4374d0faf7b64b8fddf9fcf" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 35875 } } peers { permanent_uuid: "c62570af9dd641179a669ff889e48930" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 33713 } } peers { permanent_uuid: "e90dc913ca484b06972a24d139392dd9" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 39123 } } }
I20260928 22:53:28.885651 6872 sys_catalog.cc:458] T 00000000000000000000000000000000 P c62570af9dd641179a669ff889e48930 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:53:28.886761 6858 sys_catalog.cc:455] T 00000000000000000000000000000000 P e90dc913ca484b06972a24d139392dd9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05ee4f38e4374d0faf7b64b8fddf9fcf. Latest consensus state: current_term: 1 leader_uuid: "05ee4f38e4374d0faf7b64b8fddf9fcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ee4f38e4374d0faf7b64b8fddf9fcf" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 35875 } } peers { permanent_uuid: "c62570af9dd641179a669ff889e48930" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 33713 } } peers { permanent_uuid: "e90dc913ca484b06972a24d139392dd9" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 39123 } } }
I20260928 22:53:28.886854 6858 sys_catalog.cc:458] T 00000000000000000000000000000000 P e90dc913ca484b06972a24d139392dd9 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:53:28.898348 6705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:28.881675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34430 (local address 127.16.121.190:35875)
0928 22:53:28.881782 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:28.881785 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:28.887061 (+ 5276us) server_negotiation.cc:407] Connection header received
0928 22:53:28.887162 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:28.887163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:28.887174 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:28.887205 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:28.889043 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:28.889749 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:28.890725 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:28.891035 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:28.891134 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:28.892621 (+ 1487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:28.892623 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:28.892624 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:28.892631 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:28.894306 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:28.894716 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:28.894718 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:28.894718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:28.894785 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:28.896339 (+ 1554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:28.896341 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:28.896342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:28.896506 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:28.896596 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:28.898155 (+ 1559us) server_negotiation.cc:299] Negotiation successful
0928 22:53:28.898245 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:53:28.899456 6907 mvcc.cc:205] Tried to move back new op lower bound from 7334445092383100928 to 7334445092363812864. Current Snapshot: MvccSnapshot[applied={T|T < 7334445092383100928}]
I20260928 22:53:28.899533 6886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05ee4f38e4374d0faf7b64b8fddf9fcf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "05ee4f38e4374d0faf7b64b8fddf9fcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ee4f38e4374d0faf7b64b8fddf9fcf" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 35875 } } peers { permanent_uuid: "c62570af9dd641179a669ff889e48930" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 33713 } } peers { permanent_uuid: "e90dc913ca484b06972a24d139392dd9" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 39123 } } }
I20260928 22:53:28.899641 6886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05ee4f38e4374d0faf7b64b8fddf9fcf [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:28.899878 6855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:28.879331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44230 (local address 127.16.121.188:39123)
0928 22:53:28.879386 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:28.879388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:28.884121 (+ 4733us) server_negotiation.cc:407] Connection header received
0928 22:53:28.884219 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:28.884221 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:28.884233 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:28.884266 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:28.885160 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:28.885884 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:28.888476 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:28.888789 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:28.888833 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:28.895882 (+ 7049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:28.895884 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:28.895884 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:28.895893 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:28.897861 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:28.899027 (+ 1166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:28.899029 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:28.899029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:28.899107 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:28.899315 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:28.899316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:28.899317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:28.899440 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:28.899518 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:28.899704 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 22:53:28.899783 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 22:53:28.900120 6872 sys_catalog.cc:455] T 00000000000000000000000000000000 P c62570af9dd641179a669ff889e48930 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "05ee4f38e4374d0faf7b64b8fddf9fcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ee4f38e4374d0faf7b64b8fddf9fcf" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 35875 } } peers { permanent_uuid: "c62570af9dd641179a669ff889e48930" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 33713 } } peers { permanent_uuid: "e90dc913ca484b06972a24d139392dd9" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 39123 } } }
I20260928 22:53:28.900226 6872 sys_catalog.cc:458] T 00000000000000000000000000000000 P c62570af9dd641179a669ff889e48930 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:53:28.900975 6911 mvcc.cc:205] Tried to move back new op lower bound from 7334445092383100928 to 7334445092363812864. Current Snapshot: MvccSnapshot[applied={T|T < 7334445092383100928}]
I20260928 22:53:28.901326 6781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:28.882128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58410 (local address 127.16.121.189:33713)
0928 22:53:28.882169 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:28.882171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:28.887221 (+ 5050us) server_negotiation.cc:407] Connection header received
0928 22:53:28.887324 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:28.887325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:28.887335 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:28.887363 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:28.888093 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:28.888814 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:28.895140 (+ 6326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:28.895484 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:28.895528 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:28.896995 (+ 1467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:28.896996 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:28.896996 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:28.897004 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:28.898633 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:28.900476 (+ 1843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:28.900477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:28.900478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:28.900540 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:28.900747 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:28.900748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:28.900748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:28.900873 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:28.900959 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:28.901148 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 22:53:28.901240 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 22:53:28.902876 6886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05ee4f38e4374d0faf7b64b8fddf9fcf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "05ee4f38e4374d0faf7b64b8fddf9fcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ee4f38e4374d0faf7b64b8fddf9fcf" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 35875 } } peers { permanent_uuid: "c62570af9dd641179a669ff889e48930" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 33713 } } peers { permanent_uuid: "e90dc913ca484b06972a24d139392dd9" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 39123 } } }
I20260928 22:53:28.902968 6886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05ee4f38e4374d0faf7b64b8fddf9fcf [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:28.904062 6858 sys_catalog.cc:455] T 00000000000000000000000000000000 P e90dc913ca484b06972a24d139392dd9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "05ee4f38e4374d0faf7b64b8fddf9fcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ee4f38e4374d0faf7b64b8fddf9fcf" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 35875 } } peers { permanent_uuid: "c62570af9dd641179a669ff889e48930" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 33713 } } peers { permanent_uuid: "e90dc913ca484b06972a24d139392dd9" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 39123 } } }
I20260928 22:53:28.904147 6858 sys_catalog.cc:458] T 00000000000000000000000000000000 P e90dc913ca484b06972a24d139392dd9 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:53:28.904431 6913 mvcc.cc:205] Tried to move back new op lower bound from 7334445092383100928 to 7334445092363812864. Current Snapshot: MvccSnapshot[applied={T|T < 7334445092383100928}]
I20260928 22:53:28.905478 6905 catalog_manager.cc:1347] Generated new cluster ID: 6de2a82012ec4ee1a90444259edfbe11
I20260928 22:53:28.905527 6905 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:53:28.920120 6905 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:53:28.920936 6905 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:53:28.930004 6905 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05ee4f38e4374d0faf7b64b8fddf9fcf: Generated new TSK 0
I20260928 22:53:28.931114 6905 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 22:53:28.963627 6855 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:53:28.961905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44244 (local address 127.16.121.188:39123)
0928 22:53:28.961954 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:28.961956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:28.962913 (+ 957us) server_negotiation.cc:407] Connection header received
0928 22:53:28.963020 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:28.963021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:28.963033 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:28.963098 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:28.963555 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44244: BlockingRecv error: recv got EOF from 127.0.0.1:44244 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260928 22:53:28.964463 6705 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:53:28.962334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34440 (local address 127.16.121.190:35875)
0928 22:53:28.962379 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:28.962381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:28.963949 (+ 1568us) server_negotiation.cc:407] Connection header received
0928 22:53:28.964047 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:28.964048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:28.964058 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:28.964088 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:28.964398 (+ 310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34440: BlockingRecv error: recv got EOF from 127.0.0.1:34440 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 22:53:28.965265 6781 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:53:28.962635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58414 (local address 127.16.121.189:33713)
0928 22:53:28.962676 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:28.962677 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:28.964721 (+ 2044us) server_negotiation.cc:407] Connection header received
0928 22:53:28.964832 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:28.964834 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:28.964844 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:28.964873 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:28.965206 (+ 333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58414: BlockingRecv error: recv got EOF from 127.0.0.1:58414 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 22:53:28.967732 6781 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:53:28.966995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58428 (local address 127.16.121.189:33713)
0928 22:53:28.967031 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:28.967032 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:28.967238 (+ 206us) server_negotiation.cc:407] Connection header received
0928 22:53:28.967344 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:28.967345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:28.967363 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:28.967392 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:28.967687 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58428: BlockingRecv error: recv got EOF from 127.0.0.1:58428 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 22:53:28.970733 6781 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:53:28.970014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58430 (local address 127.16.121.189:33713)
0928 22:53:28.970045 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:28.970047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:28.970266 (+ 219us) server_negotiation.cc:407] Connection header received
0928 22:53:28.970361 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:28.970362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:28.970373 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:28.970401 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:28.970688 (+ 287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58430: BlockingRecv error: recv got EOF from 127.0.0.1:58430 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 22:53:28.974856 6781 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:53:28.974109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58436 (local address 127.16.121.189:33713)
0928 22:53:28.974145 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:28.974146 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:28.974353 (+ 207us) server_negotiation.cc:407] Connection header received
0928 22:53:28.974443 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:28.974444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:28.974455 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:28.974483 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:28.974808 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58436: BlockingRecv error: recv got EOF from 127.0.0.1:58436 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 22:53:28.981000 6781 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:53:28.980142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58440 (local address 127.16.121.189:33713)
0928 22:53:28.980179 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:28.980180 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:28.980436 (+ 256us) server_negotiation.cc:407] Connection header received
0928 22:53:28.980541 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:28.980542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:28.980554 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:28.980582 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:28.980948 (+ 366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58440: BlockingRecv error: recv got EOF from 127.0.0.1:58440 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 22:53:28.991446 6781 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:53:28.990451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58446 (local address 127.16.121.189:33713)
0928 22:53:28.990497 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:28.990499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:28.990788 (+ 289us) server_negotiation.cc:407] Connection header received
0928 22:53:28.990912 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:28.990913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:28.990925 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:28.990956 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:28.991387 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58446: BlockingRecv error: recv got EOF from 127.0.0.1:58446 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 22:53:29.009925 6781 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:53:29.008946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58454 (local address 127.16.121.189:33713)
0928 22:53:29.008992 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:29.008994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:29.009288 (+ 294us) server_negotiation.cc:407] Connection header received
0928 22:53:29.009402 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:29.009403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:29.009416 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:29.009447 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:29.009863 (+ 416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58454: BlockingRecv error: recv got EOF from 127.0.0.1:58454 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 22:53:29.044373 6781 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:53:29.043459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58458 (local address 127.16.121.189:33713)
0928 22:53:29.043543 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:29.043547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:29.043728 (+ 181us) server_negotiation.cc:407] Connection header received
0928 22:53:29.043795 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:29.043796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:29.043808 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:29.043864 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:29.044304 (+ 440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58458: BlockingRecv error: recv got EOF from 127.0.0.1:58458 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260928 22:53:29.110309 6781 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:53:29.109601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58470 (local address 127.16.121.189:33713)
0928 22:53:29.109667 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:29.109668 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:29.109706 (+ 38us) server_negotiation.cc:407] Connection header received
0928 22:53:29.109776 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:29.109777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:29.109788 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:29.109858 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:29.110174 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58470: BlockingRecv error: recv got EOF from 127.0.0.1:58470 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260928 22:53:29.240531 6781 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:53:29.239659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58474 (local address 127.16.121.189:33713)
0928 22:53:29.239739 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:29.239741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:29.239752 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:53:29.239824 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:29.239825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:29.239836 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:29.239927 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:29.240449 (+ 522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58474: BlockingRecv error: recv got EOF from 127.0.0.1:58474 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260928 22:53:29.498502 6781 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:53:29.497760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58484 (local address 127.16.121.189:33713)
0928 22:53:29.497823 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:29.497825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:29.497872 (+ 47us) server_negotiation.cc:407] Connection header received
0928 22:53:29.497935 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:29.497935 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:29.497946 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:29.498008 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:29.498439 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58484: BlockingRecv error: recv got EOF from 127.0.0.1:58484 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 22:53:29.801494 6869 catalog_manager.cc:1259] Loaded cluster ID: 6de2a82012ec4ee1a90444259edfbe11
I20260928 22:53:29.801575 6869 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e90dc913ca484b06972a24d139392dd9: loading cluster ID for follower catalog manager: success
I20260928 22:53:29.802383 6869 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e90dc913ca484b06972a24d139392dd9: acquiring CA information for follower catalog manager: success
I20260928 22:53:29.802641 6869 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e90dc913ca484b06972a24d139392dd9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 22:53:29.831897 6883 catalog_manager.cc:1259] Loaded cluster ID: 6de2a82012ec4ee1a90444259edfbe11
I20260928 22:53:29.832000 6883 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c62570af9dd641179a669ff889e48930: loading cluster ID for follower catalog manager: success
I20260928 22:53:29.832885 6883 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c62570af9dd641179a669ff889e48930: acquiring CA information for follower catalog manager: success
I20260928 22:53:29.833189 6883 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c62570af9dd641179a669ff889e48930: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 22:53:30.014603 6983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:30.012025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58492 (local address 127.16.121.189:33713)
0928 22:53:30.012160 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:30.012162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:30.012172 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:53:30.012198 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:30.012198 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:30.012251 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:30.012347 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:53:30.012488 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:30.012995 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:30.013797 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:30.014004 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:30.014064 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:30.014251 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:53:30.014368 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:53:30.014427 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:30.014516 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 22:53:30.014535 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":93,"threads_started":1}
I20260928 22:53:30.019447 6989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:30.016676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44256 (local address 127.16.121.188:39123)
0928 22:53:30.016859 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:30.016861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:30.016869 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:53:30.016907 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:30.016907 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:30.016943 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:30.017018 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:53:30.017247 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:30.017733 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:30.018649 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:30.018865 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:30.018931 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:30.019140 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:53:30.019247 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:53:30.019311 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:30.019364 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 22:53:30.019382 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":109,"threads_started":1}
I20260928 22:53:30.024296 6996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:30.021300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34444 (local address 127.16.121.190:35875)
0928 22:53:30.021421 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:30.021424 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:30.021434 (+ 10us) server_negotiation.cc:407] Connection header received
0928 22:53:30.021482 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:30.021483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:30.021535 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:30.021622 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:53:30.021779 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:30.022593 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:30.023402 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:30.023639 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:30.023703 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:30.023884 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:53:30.024025 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:53:30.024088 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:30.024177 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 22:53:30.024201 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":64,"threads_started":1}
I20260928 22:53:30.024636 6983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:30.021536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58504 (local address 127.16.121.189:33713)
0928 22:53:30.021594 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:30.021596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:30.021604 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:53:30.021786 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:30.021787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:30.021800 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:30.021850 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:53:30.022002 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:30.022469 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:30.023616 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:30.023935 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:30.024006 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:30.024335 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:53:30.024382 (+ 47us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:53:30.024435 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:30.024540 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 22:53:30.024570 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 22:53:30.025113 6989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:30.021299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44258 (local address 127.16.121.188:39123)
0928 22:53:30.021334 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:30.021336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:30.021541 (+ 205us) server_negotiation.cc:407] Connection header received
0928 22:53:30.021643 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:30.021645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:30.021657 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:30.021727 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:53:30.021849 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:30.022557 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:30.024045 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:30.024319 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:30.024395 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:30.024823 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:53:30.024855 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:53:30.024902 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:30.024995 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 22:53:30.025022 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 22:53:30.026487 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 6634
I20260928 22:53:30.032526 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 6709
I20260928 22:53:30.038456 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 6784
2026-09-28T22:53:30Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2938 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 22:53:30 dist-test-slave-56vw krb5kdc[7000](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 22:53:30 dist-test-slave-56vw krb5kdc[7000](info): set up 2 sockets
Sep 28 22:53:30 dist-test-slave-56vw krb5kdc[7000](info): commencing operation
krb5kdc: starting...
W20260928 22:53:31.147516 16870 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/krb5kdc/test-user.keytab.
Sep 28 22:53:31 dist-test-slave-56vw krb5kdc[7000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T22:53:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T22:53:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.121.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
Entry for principal kudu/127.16.121.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab.
I20260928 22:53:31.275099 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:37733
--webserver_interface=127.16.121.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:43573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:37733,127.16.121.189:43481,127.16.121.188:40251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 22:53:31.357713 7013 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 22:53:31.357970 7013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:31.358009 7013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:31.358069 7013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:31.359400 7013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:53:31.359467 7013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:31.359493 7013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:31.359522 7013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:53:31.359552 7013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:53:31.361100 7013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:43573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:37733,127.16.121.189:43481,127.16.121.188:40251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.190.keytab
--principal=kudu/127.16.121.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.190:37733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.121.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:31.361379 7013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:31.361616 7013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:31.366776 7021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:31.366775 7019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:31.366863 7013 server_base.cc:1034] running on GCE node
W20260928 22:53:31.366770 7018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:31.367336 7013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:31.367620 7013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:31.368788 7013 hybrid_clock.cc:648] HybridClock initialized: now 1790636011368770 us; error 37 us; skew 500 ppm
Sep 28 22:53:31 dist-test-slave-56vw krb5kdc[7000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636011, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:31.371490 7013 init.cc:377] Logged in from keytab as kudu/127.16.121.190@KRBTEST.COM (short username kudu)
I20260928 22:53:31.371971 7013 webserver.cc:460] Webserver started at http://127.16.121.190:43417/ using document root <none> and password file <none>
I20260928 22:53:31.372217 7013 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:31.372268 7013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:31.372396 7013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:31.373512 7013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-0/data/instance:
uuid: "cb1138d8852b41a5a19a537e0d7a94b0"
format_stamp: "Formatted at 2026-09-28 22:53:31 on dist-test-slave-56vw"
server_key: "9af02e82f83f3e98df91a60200742d59"
server_key_iv: "da3882a45acfc9caaf8190a20fc660a7"
server_key_version: "encryptionkey@0"
I20260928 22:53:31.373925 7013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-0/wal/instance:
uuid: "cb1138d8852b41a5a19a537e0d7a94b0"
format_stamp: "Formatted at 2026-09-28 22:53:31 on dist-test-slave-56vw"
server_key: "9af02e82f83f3e98df91a60200742d59"
server_key_iv: "da3882a45acfc9caaf8190a20fc660a7"
server_key_version: "encryptionkey@0"
I20260928 22:53:31.375373 7013 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 22:53:31.376204 7028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:31.376484 7013 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 22:53:31.376562 7013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-0/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-0/wal
uuid: "cb1138d8852b41a5a19a537e0d7a94b0"
format_stamp: "Formatted at 2026-09-28 22:53:31 on dist-test-slave-56vw"
server_key: "9af02e82f83f3e98df91a60200742d59"
server_key_iv: "da3882a45acfc9caaf8190a20fc660a7"
server_key_version: "encryptionkey@0"
I20260928 22:53:31.376652 7013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:31.396677 7013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:31.399288 7013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:31.399484 7013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:31.403584 7013 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.190:37733
I20260928 22:53:31.403616 7080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.190:37733 every 8 connection(s)
I20260928 22:53:31.403988 7013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-0/data/info.pb
I20260928 22:53:31.404881 7081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:31.406579 7081 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37733 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:53:31.410542 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 7013
I20260928 22:53:31.410655 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-0/wal/instance
I20260928 22:53:31.410863 16870 external_mini_cluster.cc:1428] Setting key b0da04a8d21514b2f5bb8c282a5e0773
Sep 28 22:53:31 dist-test-slave-56vw krb5kdc[7000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636011, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:53:31.417537 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:31.407338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37733 (local address 127.0.0.1:41364)
0928 22:53:31.407562 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:31.407609 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:31.408152 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:31.408471 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:31.408473 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:31.408753 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:31.408914 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:31.408918 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:31.409944 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:31.409945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:31.410631 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:31.410633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:31.411427 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:31.411428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:31.411494 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:31.412117 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:31.412126 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:31.413760 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:31.416080 (+ 2320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:31.416086 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:31.416087 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:31.416277 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:31.416546 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:31.416547 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:31.416547 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:31.416623 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:31.417089 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:31.417094 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:31.417148 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 22:53:31.417219 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 22:53:31.417370 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":84,"threads_started":1}
I20260928 22:53:31.417599 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:31.407373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41364 (local address 127.16.121.190:37733)
0928 22:53:31.407573 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:31.407582 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:31.407713 (+ 131us) server_negotiation.cc:407] Connection header received
0928 22:53:31.408314 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:31.408316 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:31.408368 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:31.408466 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:31.409017 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:31.409824 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:31.410742 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:31.411302 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:31.411424 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:31.413854 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:31.413861 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:31.413861 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:31.413869 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:31.415995 (+ 2126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:31.416358 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:31.416362 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:31.416363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:31.416477 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:31.416686 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:31.416687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:31.416687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:31.417022 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:31.417115 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:31.417286 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 22:53:31.417459 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":142,"thread_start_us":76,"threads_started":1}
I20260928 22:53:31.418543 7081 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.121.189" port: 43481 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:53:31.419003 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:31.418850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.189:43481 (local address 127.0.0.1:34786)
0928 22:53:31.418918 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:31.418969 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.189:43481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 22:53:31.419188 7031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.121.189:43481: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.16.121.189@KRBTEST.COM; defaulting to no policy
I20260928 22:53:31.419678 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:31.419587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.189:43481 (local address 127.0.0.1:34802)
0928 22:53:31.419632 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:31.419659 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.189:43481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 22:53:31.419807 7081 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.121.189:43481: Network error: Client connection negotiation failed: client connection to 127.16.121.189:43481: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.189.keytab.
Entry for principal kudu/127.16.121.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.189.keytab.
I20260928 22:53:31.438095 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.189:43481
--webserver_interface=127.16.121.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.189.keytab
--principal=kudu/127.16.121.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:43573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:37733,127.16.121.189:43481,127.16.121.188:40251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:53:31.469079 7081 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.121.189" port: 43481 } attempt: 1
I20260928 22:53:31.469656 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:31.469471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.189:43481 (local address 127.0.0.1:34806)
0928 22:53:31.469535 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:31.469618 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.189:43481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 22:53:31.470017 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:31.469923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.189:43481 (local address 127.0.0.1:34818)
0928 22:53:31.469966 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:31.469991 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.189:43481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 22:53:31.470105 7081 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.121.189:43481: Network error: Client connection negotiation failed: client connection to 127.16.121.189:43481: connect: Connection refused (error 111)
W20260928 22:53:31.521356 7088 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 22:53:31.521610 7088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:31.521639 7088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:31.521682 7088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:31.522879 7088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:53:31.522948 7088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:31.522974 7088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:31.523003 7088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:53:31.523033 7088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:53:31.524647 7088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:43573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:37733,127.16.121.189:43481,127.16.121.188:40251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.189.keytab
--principal=kudu/127.16.121.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.189:43481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.121.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:31.524948 7088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:31.525188 7088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:31.530179 7093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:31.530193 7094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:31.530195 7096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:31.530228 7088 server_base.cc:1034] running on GCE node
I20260928 22:53:31.530716 7088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:31.531018 7088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:31.532220 7088 hybrid_clock.cc:648] HybridClock initialized: now 1790636011532207 us; error 34 us; skew 500 ppm
Sep 28 22:53:31 dist-test-slave-56vw krb5kdc[7000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636011, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:31.534942 7088 init.cc:377] Logged in from keytab as kudu/127.16.121.189@KRBTEST.COM (short username kudu)
I20260928 22:53:31.535439 7088 webserver.cc:460] Webserver started at http://127.16.121.189:35761/ using document root <none> and password file <none>
I20260928 22:53:31.535699 7088 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:31.535766 7088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:31.535907 7088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:31.536969 7088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-1/data/instance:
uuid: "fc986dfc94144d8ebf38873259309ba1"
format_stamp: "Formatted at 2026-09-28 22:53:31 on dist-test-slave-56vw"
server_key: "3908558f06e05b00cab7ca014112b87b"
server_key_iv: "53749af7b30a105b06eae25ce85c5634"
server_key_version: "encryptionkey@0"
I20260928 22:53:31.537314 7088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-1/wal/instance:
uuid: "fc986dfc94144d8ebf38873259309ba1"
format_stamp: "Formatted at 2026-09-28 22:53:31 on dist-test-slave-56vw"
server_key: "3908558f06e05b00cab7ca014112b87b"
server_key_iv: "53749af7b30a105b06eae25ce85c5634"
server_key_version: "encryptionkey@0"
I20260928 22:53:31.538254 7081 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.121.189" port: 43481 } attempt: 2
I20260928 22:53:31.538667 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:31.538534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.189:43481 (local address 127.0.0.1:34832)
0928 22:53:31.538595 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:31.538626 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.189:43481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 22:53:31.538694 7088 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 22:53:31.539104 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:31.538980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.189:43481 (local address 127.0.0.1:34844)
0928 22:53:31.539020 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:31.539078 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.189:43481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 22:53:31.539206 7081 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.121.189:43481: Network error: Client connection negotiation failed: client connection to 127.16.121.189:43481: connect: Connection refused (error 111)
I20260928 22:53:31.539683 7103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:31.539850 7088 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:53:31.539944 7088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-1/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-1/wal
uuid: "fc986dfc94144d8ebf38873259309ba1"
format_stamp: "Formatted at 2026-09-28 22:53:31 on dist-test-slave-56vw"
server_key: "3908558f06e05b00cab7ca014112b87b"
server_key_iv: "53749af7b30a105b06eae25ce85c5634"
server_key_version: "encryptionkey@0"
I20260928 22:53:31.540025 7088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:31.558466 7088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:31.560969 7088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:31.561139 7088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:31.565270 7088 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.189:43481
I20260928 22:53:31.565300 7155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.189:43481 every 8 connection(s)
I20260928 22:53:31.565675 7088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-1/data/info.pb
I20260928 22:53:31.566596 7156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:31.568320 7156 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37733 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:53:31.573242 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 7088
I20260928 22:53:31.573364 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-1/wal/instance
I20260928 22:53:31.573552 16870 external_mini_cluster.cc:1428] Setting key 13227fa52cca712ae09de02b6b389251
Sep 28 22:53:31 dist-test-slave-56vw krb5kdc[7000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636011, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.189@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:53:31.578306 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:31.569097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41372 (local address 127.16.121.190:37733)
0928 22:53:31.569158 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:31.569160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:31.569408 (+ 248us) server_negotiation.cc:407] Connection header received
0928 22:53:31.569920 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:31.569921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:31.569934 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:31.569975 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:31.570627 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:31.571249 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:31.572276 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:31.572488 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:31.572572 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:31.575021 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:31.575023 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:31.575024 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:31.575032 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:31.576513 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:31.576966 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:31.576968 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:31.576968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:31.577026 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:31.577475 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:31.577476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:31.577477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:31.577613 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:31.577718 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:31.578019 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 22:53:31.578108 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 22:53:31.578447 7158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:31.569080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37733 (local address 127.0.0.1:41372)
0928 22:53:31.569283 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:31.569320 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:31.569805 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:31.570064 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:31.570067 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:31.570323 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:31.570525 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:31.570529 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:31.571341 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:31.571349 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:31.572147 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:31.572149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:31.572583 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:31.572584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:31.572713 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:31.573225 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:31.573238 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:31.574914 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:31.576616 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:31.576623 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:31.576625 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:31.576876 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:31.577101 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:31.577102 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:31.577102 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:31.577217 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:31.577709 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:31.577716 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:31.577877 (+ 161us) client_negotiation.cc:769] Sending connection context
0928 22:53:31.577986 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 22:53:31.578157 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":105,"threads_started":1}
I20260928 22:53:31.579005 7156 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.121.189" port: 43481 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.121.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.121.188@KRBTEST.COM" created.
Sep 28 22:53:31 dist-test-slave-56vw krb5kdc[7000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636011, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.189@KRBTEST.COM for kudu/127.16.121.189@KRBTEST.COM
I20260928 22:53:31.587129 7158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:31.579299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.189:43481 (local address 127.0.0.1:34850)
0928 22:53:31.579351 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:31.579362 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:31.579409 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:31.579638 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:31.579638 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:31.579812 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:31.579889 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:31.579891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:31.580822 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:31.580823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:31.581371 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:31.581372 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:31.581922 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:31.581923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:31.581987 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:31.582354 (+ 367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:31.582358 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:31.583955 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:31.586046 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:31.586046 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:31.586047 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:31.586219 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:31.586434 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:31.586435 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:31.586435 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:31.586490 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:31.586779 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:31.586779 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:31.586818 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 22:53:31.586882 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 22:53:31.586989 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 22:53:31.587212 7160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:31.579343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34850 (local address 127.16.121.189:43481)
0928 22:53:31.579507 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:31.579512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:31.579534 (+ 22us) server_negotiation.cc:407] Connection header received
0928 22:53:31.579551 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:31.579552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:31.579562 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:31.579639 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:31.579968 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:31.580706 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:31.581448 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:31.581838 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:31.581924 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:31.584037 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:31.584046 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:31.584046 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:31.584053 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:31.585959 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:31.586310 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:31.586313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:31.586314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:31.586371 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:31.586555 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:31.586556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:31.586556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:31.586711 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:31.586803 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:31.586915 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 22:53:31.587080 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":139,"thread_start_us":92,"threads_started":1}
I20260928 22:53:31.587785 7156 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.121.188" port: 40251 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:53:31.588160 7158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:31.588044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:40251 (local address 127.0.0.1:41206)
0928 22:53:31.588092 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:31.588134 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.188:40251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 22:53:31.588310 7106 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.121.188:40251: connect: Connection refused (error 111)
I20260928 22:53:31.588728 7158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:31.588656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:40251 (local address 127.0.0.1:41208)
0928 22:53:31.588688 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:31.588712 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.188:40251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260928 22:53:31.588850 7156 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.121.188:40251: Network error: Client connection negotiation failed: client connection to 127.16.121.188:40251: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.121.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.188.keytab.
Entry for principal kudu/127.16.121.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.188.keytab.
I20260928 22:53:31.602394 16870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
/tmp/dist-test-taskhFq4Cy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.188:40251
--webserver_interface=127.16.121.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.188.keytab
--principal=kudu/127.16.121.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.121.148:43573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.121.190:37733,127.16.121.189:43481,127.16.121.188:40251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 22:53:31.630367 7081 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.121.189" port: 43481 } attempt: 3
Sep 28 22:53:31 dist-test-slave-56vw krb5kdc[7000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636011, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for kudu/127.16.121.189@KRBTEST.COM
I20260928 22:53:31.637928 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:31.630791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.189:43481 (local address 127.0.0.1:34852)
0928 22:53:31.630839 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:31.630853 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:31.630927 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:31.631098 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:31.631099 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:31.631324 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:31.631400 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:31.631402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:31.632123 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:31.632125 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:31.632671 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:31.632672 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:31.633072 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:31.633073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:31.633157 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:31.633658 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:31.633663 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:31.635151 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:31.636868 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:31.636869 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:31.636870 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:31.637050 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:31.637257 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:31.637258 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:31.637258 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:31.637323 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:31.637576 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:31.637578 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:31.637625 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 22:53:31.637706 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 22:53:31.637825 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 22:53:31.637928 7160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:31.630852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34852 (local address 127.16.121.189:43481)
0928 22:53:31.630924 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:31.630927 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:31.630938 (+ 11us) server_negotiation.cc:407] Connection header received
0928 22:53:31.630991 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:31.630992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:31.631009 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:31.631078 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:31.631496 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:31.632026 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:31.632767 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:31.632997 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:31.633075 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:31.635244 (+ 2169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:31.635246 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:31.635246 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:31.635256 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:31.636733 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:31.637123 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:31.637125 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:31.637125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:31.637184 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:31.637384 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:31.637385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:31.637386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:31.637504 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:31.637573 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:31.637730 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 22:53:31.637826 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 22:53:31.638067 7156 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.121.188" port: 40251 } attempt: 1
I20260928 22:53:31.638433 7158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:31.638318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:40251 (local address 127.0.0.1:41216)
0928 22:53:31.638378 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:31.638405 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.188:40251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 22:53:31.638751 7158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:31.638658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:40251 (local address 127.0.0.1:41230)
0928 22:53:31.638707 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:31.638728 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.188:40251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 22:53:31.638796 7081 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.121.188" port: 40251 } has no permanent_uuid. Determining permanent_uuid...
W20260928 22:53:31.638870 7156 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.121.188:40251: Network error: Client connection negotiation failed: client connection to 127.16.121.188:40251: connect: Connection refused (error 111)
I20260928 22:53:31.639145 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:31.639029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:40251 (local address 127.0.0.1:41236)
0928 22:53:31.639094 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:31.639120 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.188:40251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 22:53:31.639437 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:31.639351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:40251 (local address 127.0.0.1:41248)
0928 22:53:31.639396 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:31.639417 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.188:40251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 22:53:31.639521 7081 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.121.188:40251: Network error: Client connection negotiation failed: client connection to 127.16.121.188:40251: connect: Connection refused (error 111)
I20260928 22:53:31.670681 7081 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.121.188" port: 40251 } attempt: 1
I20260928 22:53:31.671326 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:31.671112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:40251 (local address 127.0.0.1:41250)
0928 22:53:31.671242 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:31.671282 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.188:40251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 22:53:31.671677 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:31.671583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:40251 (local address 127.0.0.1:41258)
0928 22:53:31.671626 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:31.671655 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.188:40251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 22:53:31.671766 7081 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.121.188:40251: Network error: Client connection negotiation failed: client connection to 127.16.121.188:40251: connect: Connection refused (error 111)
W20260928 22:53:31.689297 7163 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 22:53:31.689520 7163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 22:53:31.689538 7163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 22:53:31.689559 7163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 22:53:31.690778 7163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 22:53:31.690819 7163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 22:53:31.690833 7163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 22:53:31.690846 7163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 22:53:31.690889 7163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 22:53:31.692529 7163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.121.148:43573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.121.190:37733,127.16.121.189:43481,127.16.121.188:40251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/krb5kdc/kudu_127.16.121.188.keytab
--principal=kudu/127.16.121.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.121.188:40251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.121.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:01:43 UTC on ccycloud.quasar-wpwwge.root.comops.site
build id 4678935
I20260928 22:53:31.692794 7163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 22:53:31.693046 7163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 22:53:31.699118 7171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 22:53:31.699138 7168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:31.699194 7163 server_base.cc:1034] running on GCE node
W20260928 22:53:31.699157 7169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 22:53:31.699617 7163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 22:53:31.699957 7163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 22:53:31.701149 7163 hybrid_clock.cc:648] HybridClock initialized: now 1790636011701138 us; error 47 us; skew 500 ppm
Sep 28 22:53:31 dist-test-slave-56vw krb5kdc[7000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636011, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 22:53:31.704041 7163 init.cc:377] Logged in from keytab as kudu/127.16.121.188@KRBTEST.COM (short username kudu)
I20260928 22:53:31.704619 7163 webserver.cc:460] Webserver started at http://127.16.121.188:42691/ using document root <none> and password file <none>
I20260928 22:53:31.704867 7163 fs_manager.cc:359] Metadata directory not provided
I20260928 22:53:31.704932 7163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 22:53:31.705072 7163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 22:53:31.706224 7163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-2/data/instance:
uuid: "2a9a565db5ad4ba38312d27b9ab193b7"
format_stamp: "Formatted at 2026-09-28 22:53:31 on dist-test-slave-56vw"
server_key: "a4b6b202e7e1c91ff51dfdce07e769b7"
server_key_iv: "d68b44cab58e54bedc9ff2c0e7ec178d"
server_key_version: "encryptionkey@0"
I20260928 22:53:31.706583 7163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-2/wal/instance:
uuid: "2a9a565db5ad4ba38312d27b9ab193b7"
format_stamp: "Formatted at 2026-09-28 22:53:31 on dist-test-slave-56vw"
server_key: "a4b6b202e7e1c91ff51dfdce07e769b7"
server_key_iv: "d68b44cab58e54bedc9ff2c0e7ec178d"
server_key_version: "encryptionkey@0"
I20260928 22:53:31.707020 7156 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.121.188" port: 40251 } attempt: 2
I20260928 22:53:31.707460 7158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:31.707340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:40251 (local address 127.0.0.1:41274)
0928 22:53:31.707396 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:31.707433 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.188:40251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 22:53:31.707782 7158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:31.707688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:40251 (local address 127.0.0.1:41282)
0928 22:53:31.707730 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:31.707765 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.121.188:40251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 22:53:31.707864 7156 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.121.188:40251: Network error: Client connection negotiation failed: client connection to 127.16.121.188:40251: connect: Connection refused (error 111)
I20260928 22:53:31.708216 7163 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 22:53:31.708954 7178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 22:53:31.709149 7163 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 22:53:31.709249 7163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-2/data,/tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-2/wal
uuid: "2a9a565db5ad4ba38312d27b9ab193b7"
format_stamp: "Formatted at 2026-09-28 22:53:31 on dist-test-slave-56vw"
server_key: "a4b6b202e7e1c91ff51dfdce07e769b7"
server_key_iv: "d68b44cab58e54bedc9ff2c0e7ec178d"
server_key_version: "encryptionkey@0"
I20260928 22:53:31.709334 7163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 22:53:31.728444 7163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 22:53:31.730609 7163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 22:53:31.730793 7163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 22:53:31.734997 7163 rpc_server.cc:307] RPC server started. Bound to: 127.16.121.188:40251
I20260928 22:53:31.735039 7230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.121.188:40251 every 8 connection(s)
I20260928 22:53:31.735615 7163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-2/data/info.pb
I20260928 22:53:31.736371 7231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 22:53:31.737572 16870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu as pid 7163
I20260928 22:53:31.737694 16870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhFq4Cy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790635733329745-16870-0/minicluster-data/master-2/wal/instance
I20260928 22:53:31.737877 16870 external_mini_cluster.cc:1428] Setting key 8e9c9828cdcbe335df37d7e42dcd439d
I20260928 22:53:31.738230 7231 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37733 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 22:53:31 dist-test-slave-56vw krb5kdc[7000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636011, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.188@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
Sep 28 22:53:31 dist-test-slave-56vw krb5kdc[7000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.121.188@KRBTEST.COM
I20260928 22:53:31.746915 7081 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.121.188" port: 40251 } attempt: 2
I20260928 22:53:31.748441 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:31.739068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41388 (local address 127.16.121.190:37733)
0928 22:53:31.739116 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:31.739118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:31.739261 (+ 143us) server_negotiation.cc:407] Connection header received
0928 22:53:31.739463 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:31.739465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:31.739477 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:31.739540 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:31.740140 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:31.740600 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:31.741893 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:31.742244 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:31.742325 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:31.744988 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:31.744989 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:31.744990 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:31.744998 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:31.746673 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:31.747323 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:31.747324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:31.747325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:31.747387 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:31.747667 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:31.747668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:31.747668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:31.747832 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:31.747893 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:31.748268 (+ 375us) server_negotiation.cc:299] Negotiation successful
0928 22:53:31.748350 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 22:53:31.748605 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:31.738952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.190:37733 (local address 127.0.0.1:41388)
0928 22:53:31.739118 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:31.739167 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:31.739362 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:31.739559 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:31.739562 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:31.739861 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:31.740025 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:31.740029 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:31.740679 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:31.740681 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:31.741812 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:31.741814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:31.742347 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:31.742349 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:31.742487 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:31.743038 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:31.743074 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:31.744888 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:31.746770 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:31.746778 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:31.746780 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:31.747213 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:31.747466 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:31.747466 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:31.747467 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:31.747609 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:31.747917 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:31.747922 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:31.748101 (+ 179us) client_negotiation.cc:769] Sending connection context
0928 22:53:31.748212 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 22:53:31.748377 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":92,"thread_start_us":80,"threads_started":1}
I20260928 22:53:31.749204 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:31.738424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41298 (local address 127.16.121.188:40251)
0928 22:53:31.738625 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:31.738635 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:31.738674 (+ 39us) server_negotiation.cc:407] Connection header received
0928 22:53:31.739362 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:31.739372 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:31.739398 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:31.739494 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:31.740119 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:31.741347 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:31.742202 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:31.742594 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:31.742712 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:31.745453 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:31.745460 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:31.745460 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:31.745468 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:31.747339 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:31.747747 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:31.747750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:31.747750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:31.747800 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:31.748007 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:31.748008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:31.748008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:31.748156 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:31.748334 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:31.749025 (+ 691us) server_negotiation.cc:299] Negotiation successful
0928 22:53:31.749101 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":148,"thread_start_us":95,"threads_started":1}
I20260928 22:53:31.749202 7231 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.121.189" port: 43481 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 22:53:31 dist-test-slave-56vw krb5kdc[7000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636011, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.190@KRBTEST.COM for kudu/127.16.121.188@KRBTEST.COM
Sep 28 22:53:31 dist-test-slave-56vw krb5kdc[7000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636011, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.188@KRBTEST.COM for kudu/127.16.121.189@KRBTEST.COM
Sep 28 22:53:31 dist-test-slave-56vw krb5kdc[7000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.121.190@KRBTEST.COM
I20260928 22:53:31.755429 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:31.747244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:40251 (local address 127.0.0.1:41308)
0928 22:53:31.747313 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:31.747323 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:31.747357 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:31.747607 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:31.747608 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:31.747775 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:31.747824 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:31.747826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:31.748736 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:31.748738 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:31.749501 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:31.749504 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:31.750022 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:31.750023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:31.750094 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:31.750512 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:31.750518 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:31.751854 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:31.754117 (+ 2263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:31.754118 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:31.754119 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:31.754473 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:31.754760 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:31.754761 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:31.754761 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:31.754828 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:31.755137 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:31.755138 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:31.755188 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 22:53:31.755263 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 22:53:31.755357 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 22:53:31.755890 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:31.747318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41308 (local address 127.16.121.188:40251)
0928 22:53:31.747476 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:31.747477 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:31.747484 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:53:31.747501 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:31.747501 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:31.747514 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:31.747578 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:31.747927 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:31.748621 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:31.749596 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:31.749900 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:31.749974 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:31.751951 (+ 1977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:31.751952 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:31.751953 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:31.751959 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:31.753991 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:31.754579 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:31.754580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:31.754580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:31.754644 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:31.754901 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:31.754902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:31.754902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:31.755034 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:31.755147 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:31.755658 (+ 511us) server_negotiation.cc:299] Negotiation successful
0928 22:53:31.755754 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":102,"threads_started":1}
I20260928 22:53:31.758205 7081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb1138d8852b41a5a19a537e0d7a94b0: Bootstrap starting.
I20260928 22:53:31.758430 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:31.749468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.189:43481 (local address 127.0.0.1:34856)
0928 22:53:31.749507 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:31.749518 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:31.749554 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:31.749743 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:31.749744 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:31.749890 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:31.749942 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:31.749943 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:31.750776 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:31.750795 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:31.751728 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:31.751730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:31.752120 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:31.752121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:31.752193 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:31.753490 (+ 1297us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:31.753494 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:31.754788 (+ 1294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:31.757159 (+ 2371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:31.757160 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:31.757160 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:31.757387 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:31.757666 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:31.757667 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:31.757668 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:31.757772 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:31.758111 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:31.758112 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:31.758160 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 22:53:31.758229 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 22:53:31.758336 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 22:53:31.758430 7160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:31.749575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34856 (local address 127.16.121.189:43481)
0928 22:53:31.749615 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:31.749617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:31.749624 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:53:31.749659 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:31.749660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:31.749670 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:31.749735 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:31.750037 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:31.750690 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:31.751812 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:31.752039 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:31.752121 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:31.754905 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:31.754907 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:31.754907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:31.754915 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:31.757024 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:31.757483 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:31.757485 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:31.757486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:31.757571 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:31.757851 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:31.757853 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:31.757853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:31.757999 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:31.758081 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:31.758257 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 22:53:31.758336 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:53:31.758757 7231 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.121.188" port: 40251 } has no permanent_uuid. Determining permanent_uuid...
I20260928 22:53:31.758970 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:31.750023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41394 (local address 127.16.121.190:37733)
0928 22:53:31.750064 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:31.750066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:31.750073 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:53:31.750116 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:31.750117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:31.750127 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:31.750167 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:31.750822 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:31.751546 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:31.752494 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:31.752789 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:31.752856 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:31.755713 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:31.755714 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:31.755715 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:31.755724 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:31.757568 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:31.757916 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:31.757917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:31.757917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:31.757975 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:31.758169 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:31.758169 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:31.758170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:31.758306 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:31.758466 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:31.758782 (+ 316us) server_negotiation.cc:299] Negotiation successful
0928 22:53:31.758857 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 22:53:31.759119 7081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb1138d8852b41a5a19a537e0d7a94b0: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:31.759497 7081 log.cc:824] T 00000000000000000000000000000000 P cb1138d8852b41a5a19a537e0d7a94b0: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:31.760327 7081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb1138d8852b41a5a19a537e0d7a94b0: No bootstrap required, opened a new log
I20260928 22:53:31.762353 7081 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb1138d8852b41a5a19a537e0d7a94b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb1138d8852b41a5a19a537e0d7a94b0" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37733 } } peers { permanent_uuid: "fc986dfc94144d8ebf38873259309ba1" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 43481 } } peers { permanent_uuid: "2a9a565db5ad4ba38312d27b9ab193b7" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 40251 } }
I20260928 22:53:31.762528 7081 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb1138d8852b41a5a19a537e0d7a94b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb1138d8852b41a5a19a537e0d7a94b0, State: Initialized, Role: FOLLOWER
I20260928 22:53:31.762629 7081 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb1138d8852b41a5a19a537e0d7a94b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb1138d8852b41a5a19a537e0d7a94b0" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37733 } } peers { permanent_uuid: "fc986dfc94144d8ebf38873259309ba1" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 43481 } } peers { permanent_uuid: "2a9a565db5ad4ba38312d27b9ab193b7" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 40251 } }
I20260928 22:53:31.762955 7238 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb1138d8852b41a5a19a537e0d7a94b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb1138d8852b41a5a19a537e0d7a94b0" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37733 } } peers { permanent_uuid: "fc986dfc94144d8ebf38873259309ba1" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 43481 } } peers { permanent_uuid: "2a9a565db5ad4ba38312d27b9ab193b7" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 40251 } } }
I20260928 22:53:31.763027 7081 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb1138d8852b41a5a19a537e0d7a94b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:53:31.763135 7238 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb1138d8852b41a5a19a537e0d7a94b0 [sys.catalog]: This master's current role is: FOLLOWER
Sep 28 22:53:31 dist-test-slave-56vw krb5kdc[7000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636011, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.188@KRBTEST.COM for kudu/127.16.121.188@KRBTEST.COM
Sep 28 22:53:31 dist-test-slave-56vw krb5kdc[7000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.121.189@KRBTEST.COM
I20260928 22:53:31.767460 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:31.759015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:40251 (local address 127.0.0.1:41312)
0928 22:53:31.759216 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:31.759225 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:31.759275 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:31.759407 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:31.759408 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:31.759559 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:31.759599 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:31.759601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:31.760496 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:31.760497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:31.761242 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:31.761245 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:31.761688 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:31.761689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:31.761762 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:31.762294 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:31.762300 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:31.764098 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:31.766220 (+ 2122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:31.766221 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:31.766222 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:31.766466 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:31.766679 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:31.766679 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:31.766680 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:31.766755 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:31.767090 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:31.767091 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:31.767137 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 22:53:31.767204 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 22:53:31.767320 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183}
I20260928 22:53:31.767870 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:31.759155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41312 (local address 127.16.121.188:40251)
0928 22:53:31.759216 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:31.759219 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:31.759324 (+ 105us) server_negotiation.cc:407] Connection header received
0928 22:53:31.759344 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:31.759344 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:31.759350 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:31.759405 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:31.759677 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:31.760411 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:31.761330 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:31.761622 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:31.761691 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:31.764200 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:31.764202 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:31.764202 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:31.764209 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:31.766102 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:31.766551 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:31.766551 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:31.766552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:31.766612 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:31.766823 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:31.766823 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:31.766824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:31.766976 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:31.767039 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:31.767642 (+ 603us) server_negotiation.cc:299] Negotiation successful
0928 22:53:31.767750 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260928 22:53:31.768266 7249 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cb1138d8852b41a5a19a537e0d7a94b0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:53:31.768333 7249 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:53:31.768991 7160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:31.759527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34862 (local address 127.16.121.189:43481)
0928 22:53:31.759563 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:31.759564 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:31.759571 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:53:31.759625 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:31.759626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:31.759635 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:31.759681 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:31.760371 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:31.760912 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:31.761595 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:31.761951 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:31.762027 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:31.765126 (+ 3099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:31.765128 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:31.765128 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:31.765136 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:31.767085 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:31.767494 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:31.767496 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:31.767496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:31.767553 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:31.768231 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:31.768232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:31.768232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:31.768369 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:31.768494 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:31.768815 (+ 321us) server_negotiation.cc:299] Negotiation successful
0928 22:53:31.768895 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:53:31.765882 7013 master_runner.cc:186] Error getting master registration for 127.16.121.189:43481: OK, instance_id { permanent_uuid: "fc986dfc94144d8ebf38873259309ba1" instance_seqno: 1790636011564073 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 22:53:31.769117 7013 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 22:53:31.770323 7231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a9a565db5ad4ba38312d27b9ab193b7: Bootstrap starting.
I20260928 22:53:31.771092 7231 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a9a565db5ad4ba38312d27b9ab193b7: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:31.771418 7231 log.cc:824] T 00000000000000000000000000000000 P 2a9a565db5ad4ba38312d27b9ab193b7: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:31.772310 7231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a9a565db5ad4ba38312d27b9ab193b7: No bootstrap required, opened a new log
I20260928 22:53:31.773336 16870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 22:53:31.774086 7231 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a9a565db5ad4ba38312d27b9ab193b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb1138d8852b41a5a19a537e0d7a94b0" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37733 } } peers { permanent_uuid: "fc986dfc94144d8ebf38873259309ba1" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 43481 } } peers { permanent_uuid: "2a9a565db5ad4ba38312d27b9ab193b7" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 40251 } }
I20260928 22:53:31.774227 7231 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a9a565db5ad4ba38312d27b9ab193b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a9a565db5ad4ba38312d27b9ab193b7, State: Initialized, Role: FOLLOWER
I20260928 22:53:31.774313 7231 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a9a565db5ad4ba38312d27b9ab193b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb1138d8852b41a5a19a537e0d7a94b0" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37733 } } peers { permanent_uuid: "fc986dfc94144d8ebf38873259309ba1" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 43481 } } peers { permanent_uuid: "2a9a565db5ad4ba38312d27b9ab193b7" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 40251 } }
I20260928 22:53:31.774617 7231 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a9a565db5ad4ba38312d27b9ab193b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:53:31.775116 7055 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "cb1138d8852b41a5a19a537e0d7a94b0"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41394
I20260928 22:53:31.775255 7055 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb1138d8852b41a5a19a537e0d7a94b0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 22:53:31.775305 7055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb1138d8852b41a5a19a537e0d7a94b0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:31.776306 7055 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb1138d8852b41a5a19a537e0d7a94b0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb1138d8852b41a5a19a537e0d7a94b0" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37733 } } peers { permanent_uuid: "fc986dfc94144d8ebf38873259309ba1" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 43481 } } peers { permanent_uuid: "2a9a565db5ad4ba38312d27b9ab193b7" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 40251 } }
I20260928 22:53:31.776396 7252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a9a565db5ad4ba38312d27b9ab193b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb1138d8852b41a5a19a537e0d7a94b0" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37733 } } peers { permanent_uuid: "fc986dfc94144d8ebf38873259309ba1" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 43481 } } peers { permanent_uuid: "2a9a565db5ad4ba38312d27b9ab193b7" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 40251 } } }
I20260928 22:53:31.776469 7252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a9a565db5ad4ba38312d27b9ab193b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:53:31.776569 7055 leader_election.cc:288] T 00000000000000000000000000000000 P cb1138d8852b41a5a19a537e0d7a94b0 [CANDIDATE]: Term 1 election: Requested vote from peers fc986dfc94144d8ebf38873259309ba1 (127.16.121.189:43481), 2a9a565db5ad4ba38312d27b9ab193b7 (127.16.121.188:40251)
W20260928 22:53:31.776985 7263 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2a9a565db5ad4ba38312d27b9ab193b7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260928 22:53:31.777077 7205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cb1138d8852b41a5a19a537e0d7a94b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2a9a565db5ad4ba38312d27b9ab193b7"
I20260928 22:53:31.777181 7205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a9a565db5ad4ba38312d27b9ab193b7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:31.779589 7205 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2a9a565db5ad4ba38312d27b9ab193b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb1138d8852b41a5a19a537e0d7a94b0 in term 1.
W20260928 22:53:31.777122 7263 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:53:31.779857 7031 leader_election.cc:302] T 00000000000000000000000000000000 P cb1138d8852b41a5a19a537e0d7a94b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a9a565db5ad4ba38312d27b9ab193b7, cb1138d8852b41a5a19a537e0d7a94b0; no voters:
I20260928 22:53:31.780014 7238 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb1138d8852b41a5a19a537e0d7a94b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 22:53:31.780181 7238 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb1138d8852b41a5a19a537e0d7a94b0 [term 1 LEADER]: Becoming Leader. State: Replica: cb1138d8852b41a5a19a537e0d7a94b0, State: Running, Role: LEADER
I20260928 22:53:31.780300 7238 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb1138d8852b41a5a19a537e0d7a94b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb1138d8852b41a5a19a537e0d7a94b0" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37733 } } peers { permanent_uuid: "fc986dfc94144d8ebf38873259309ba1" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 43481 } } peers { permanent_uuid: "2a9a565db5ad4ba38312d27b9ab193b7" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 40251 } }
I20260928 22:53:31.780723 7238 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb1138d8852b41a5a19a537e0d7a94b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb1138d8852b41a5a19a537e0d7a94b0. Latest consensus state: current_term: 1 leader_uuid: "cb1138d8852b41a5a19a537e0d7a94b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb1138d8852b41a5a19a537e0d7a94b0" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37733 } } peers { permanent_uuid: "fc986dfc94144d8ebf38873259309ba1" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 43481 } } peers { permanent_uuid: "2a9a565db5ad4ba38312d27b9ab193b7" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 40251 } } }
I20260928 22:53:31.780813 7238 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb1138d8852b41a5a19a537e0d7a94b0 [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:31.780157 7163 master_runner.cc:186] Error getting master registration for 127.16.121.189:43481: OK, instance_id { permanent_uuid: "fc986dfc94144d8ebf38873259309ba1" instance_seqno: 1790636011564073 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 22:53:31.782437 7163 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 22:53:31.787693 7271 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 22:53:31.787949 7271 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 22:53:31.794677 7160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:31.783409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34872 (local address 127.16.121.189:43481)
0928 22:53:31.783451 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:31.783465 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:31.784066 (+ 601us) server_negotiation.cc:407] Connection header received
0928 22:53:31.784200 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:31.784202 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:31.784214 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:31.784246 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:31.785478 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:31.786232 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:31.787878 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:31.788663 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:31.788717 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:31.791258 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:31.791260 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:31.791260 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:31.791286 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:31.792960 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:31.793545 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:31.793546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:31.793547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:31.793618 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:31.793928 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:31.793929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:31.793929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:31.794079 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:31.794163 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:31.794507 (+ 344us) server_negotiation.cc:299] Negotiation successful
0928 22:53:31.794589 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 22:53:31.796815 7205 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2a9a565db5ad4ba38312d27b9ab193b7 [term 1 FOLLOWER]: Refusing update from remote peer cb1138d8852b41a5a19a537e0d7a94b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:53:31.797165 7269 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cb1138d8852b41a5a19a537e0d7a94b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a9a565db5ad4ba38312d27b9ab193b7" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 40251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:53:31.798691 7252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a9a565db5ad4ba38312d27b9ab193b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb1138d8852b41a5a19a537e0d7a94b0. Latest consensus state: current_term: 1 leader_uuid: "cb1138d8852b41a5a19a537e0d7a94b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb1138d8852b41a5a19a537e0d7a94b0" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37733 } } peers { permanent_uuid: "fc986dfc94144d8ebf38873259309ba1" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 43481 } } peers { permanent_uuid: "2a9a565db5ad4ba38312d27b9ab193b7" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 40251 } } }
I20260928 22:53:31.798801 7252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a9a565db5ad4ba38312d27b9ab193b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:53:31.799021 7156 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.121.188" port: 40251 } attempt: 3
I20260928 22:53:31.799486 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:31.783773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41402 (local address 127.16.121.190:37733)
0928 22:53:31.783811 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:31.783813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:31.784034 (+ 221us) server_negotiation.cc:407] Connection header received
0928 22:53:31.784144 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:31.784146 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:31.784157 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:31.784190 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:31.786264 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:31.787098 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:31.789974 (+ 2876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:31.790321 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:31.790363 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:31.795128 (+ 4765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:31.795130 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:31.795130 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:31.795139 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:31.797220 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:31.798017 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:31.798018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:31.798019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:31.798081 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:31.798462 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:31.798463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:31.798463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:31.798625 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:31.798703 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:31.799300 (+ 597us) server_negotiation.cc:299] Negotiation successful
0928 22:53:31.799390 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 22:53:31.803263 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:31.783631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41318 (local address 127.16.121.188:40251)
0928 22:53:31.783681 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:31.783683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:31.785285 (+ 1602us) server_negotiation.cc:407] Connection header received
0928 22:53:31.785398 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:31.785399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:31.785410 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:31.785442 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:31.787902 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:31.788721 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:31.793195 (+ 4474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:31.793552 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:31.793596 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:31.795639 (+ 2043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:31.795641 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:31.795641 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:31.795649 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:31.798027 (+ 2378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:31.798476 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:31.798477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:31.798478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:31.798539 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:31.798759 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:31.798760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:31.798760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:31.798892 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:31.798996 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:31.803027 (+ 4031us) server_negotiation.cc:299] Negotiation successful
0928 22:53:31.803141 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
Sep 28 22:53:31 dist-test-slave-56vw krb5kdc[7000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790636011, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.121.189@KRBTEST.COM for kudu/127.16.121.188@KRBTEST.COM
I20260928 22:53:31.807456 7269 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb1138d8852b41a5a19a537e0d7a94b0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cb1138d8852b41a5a19a537e0d7a94b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb1138d8852b41a5a19a537e0d7a94b0" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37733 } } peers { permanent_uuid: "fc986dfc94144d8ebf38873259309ba1" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 43481 } } peers { permanent_uuid: "2a9a565db5ad4ba38312d27b9ab193b7" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 40251 } } }
I20260928 22:53:31.807575 7269 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb1138d8852b41a5a19a537e0d7a94b0 [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:31.807971 7252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a9a565db5ad4ba38312d27b9ab193b7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cb1138d8852b41a5a19a537e0d7a94b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb1138d8852b41a5a19a537e0d7a94b0" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37733 } } peers { permanent_uuid: "fc986dfc94144d8ebf38873259309ba1" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 43481 } } peers { permanent_uuid: "2a9a565db5ad4ba38312d27b9ab193b7" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 40251 } } }
I20260928 22:53:31.808063 7252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a9a565db5ad4ba38312d27b9ab193b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:53:31.809001 7158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:31.799289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.121.188:40251 (local address 127.0.0.1:41324)
0928 22:53:31.799373 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0928 22:53:31.799385 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 22:53:31.799421 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 22:53:31.799656 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 22:53:31.799657 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 22:53:31.799831 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 22:53:31.799876 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:31.799878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:31.800931 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:31.800932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:31.801840 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 22:53:31.801842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 22:53:31.802382 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 22:53:31.802383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 22:53:31.802464 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:31.802989 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 22:53:31.802994 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 22:53:31.804516 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 22:53:31.806787 (+ 2271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:31.806788 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:31.806789 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:31.807030 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:31.808269 (+ 1239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 22:53:31.808270 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 22:53:31.808270 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 22:53:31.808338 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 22:53:31.808647 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 22:53:31.808648 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 22:53:31.808696 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 22:53:31.808766 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 22:53:31.808889 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 22:53:31.809373 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:31.799491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41324 (local address 127.16.121.188:40251)
0928 22:53:31.799529 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:31.799531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:31.799538 (+ 7us) server_negotiation.cc:407] Connection header received
0928 22:53:31.799557 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:31.799558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:31.799567 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:31.799616 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:31.799959 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:31.800820 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:31.801943 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:31.802305 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:31.802371 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:31.804591 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 22:53:31.804593 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 22:53:31.804593 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 22:53:31.804601 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 22:53:31.806679 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:31.808106 (+ 1427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:31.808107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:31.808108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:31.808169 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 22:53:31.808423 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 22:53:31.808424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 22:53:31.808424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 22:53:31.808556 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 22:53:31.808620 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:31.809185 (+ 565us) server_negotiation.cc:299] Negotiation successful
0928 22:53:31.809279 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 22:53:31.809438 7271 catalog_manager.cc:1347] Generated new cluster ID: aa8b0680d11844a7ac5adf3b8510e2ce
I20260928 22:53:31.809491 7271 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 22:53:31.810971 7156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc986dfc94144d8ebf38873259309ba1: Bootstrap starting.
I20260928 22:53:31.811825 7156 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc986dfc94144d8ebf38873259309ba1: Neither blocks nor log segments found. Creating new log.
I20260928 22:53:31.812140 7156 log.cc:824] T 00000000000000000000000000000000 P fc986dfc94144d8ebf38873259309ba1: Log is configured to *not* fsync() on all Append() calls
I20260928 22:53:31.812511 7276 mvcc.cc:205] Tried to move back new op lower bound from 7334445104312791040 to 7334445104252813312. Current Snapshot: MvccSnapshot[applied={T|T < 7334445104312791040}]
I20260928 22:53:31.823904 7156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc986dfc94144d8ebf38873259309ba1: No bootstrap required, opened a new log
I20260928 22:53:31.826201 7156 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc986dfc94144d8ebf38873259309ba1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb1138d8852b41a5a19a537e0d7a94b0" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37733 } } peers { permanent_uuid: "fc986dfc94144d8ebf38873259309ba1" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 43481 } } peers { permanent_uuid: "2a9a565db5ad4ba38312d27b9ab193b7" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 40251 } }
I20260928 22:53:31.826388 7156 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc986dfc94144d8ebf38873259309ba1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc986dfc94144d8ebf38873259309ba1, State: Initialized, Role: FOLLOWER
I20260928 22:53:31.826481 7156 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc986dfc94144d8ebf38873259309ba1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb1138d8852b41a5a19a537e0d7a94b0" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37733 } } peers { permanent_uuid: "fc986dfc94144d8ebf38873259309ba1" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 43481 } } peers { permanent_uuid: "2a9a565db5ad4ba38312d27b9ab193b7" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 40251 } }
I20260928 22:53:31.826855 7156 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc986dfc94144d8ebf38873259309ba1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 22:53:31.828333 7129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc986dfc94144d8ebf38873259309ba1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 22:53:31.829344 7129 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fc986dfc94144d8ebf38873259309ba1 [term 1 FOLLOWER]: Refusing update from remote peer cb1138d8852b41a5a19a537e0d7a94b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 22:53:31.829564 7130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cb1138d8852b41a5a19a537e0d7a94b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fc986dfc94144d8ebf38873259309ba1"
I20260928 22:53:31.829720 7238 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cb1138d8852b41a5a19a537e0d7a94b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc986dfc94144d8ebf38873259309ba1" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 43481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 22:53:31.830427 7130 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fc986dfc94144d8ebf38873259309ba1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb1138d8852b41a5a19a537e0d7a94b0 in term 1.
I20260928 22:53:31.833513 7296 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc986dfc94144d8ebf38873259309ba1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb1138d8852b41a5a19a537e0d7a94b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb1138d8852b41a5a19a537e0d7a94b0" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37733 } } peers { permanent_uuid: "fc986dfc94144d8ebf38873259309ba1" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 43481 } } peers { permanent_uuid: "2a9a565db5ad4ba38312d27b9ab193b7" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 40251 } } }
I20260928 22:53:31.833616 7296 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc986dfc94144d8ebf38873259309ba1 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:53:31.833669 7296 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc986dfc94144d8ebf38873259309ba1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb1138d8852b41a5a19a537e0d7a94b0. Latest consensus state: current_term: 1 leader_uuid: "cb1138d8852b41a5a19a537e0d7a94b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb1138d8852b41a5a19a537e0d7a94b0" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37733 } } peers { permanent_uuid: "fc986dfc94144d8ebf38873259309ba1" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 43481 } } peers { permanent_uuid: "2a9a565db5ad4ba38312d27b9ab193b7" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 40251 } } }
I20260928 22:53:31.833719 7296 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc986dfc94144d8ebf38873259309ba1 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 22:53:31.833750 7296 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc986dfc94144d8ebf38873259309ba1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cb1138d8852b41a5a19a537e0d7a94b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb1138d8852b41a5a19a537e0d7a94b0" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37733 } } peers { permanent_uuid: "fc986dfc94144d8ebf38873259309ba1" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 43481 } } peers { permanent_uuid: "2a9a565db5ad4ba38312d27b9ab193b7" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 40251 } } }
I20260928 22:53:31.833791 7296 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc986dfc94144d8ebf38873259309ba1 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 22:53:31.835952 7292 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fc986dfc94144d8ebf38873259309ba1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 22:53:31.836021 7292 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 22:53:31.836458 7294 mvcc.cc:205] Tried to move back new op lower bound from 7334445104312791040 to 7334445104252813312. Current Snapshot: MvccSnapshot[applied={T|T < 7334445104312791040}]
I20260928 22:53:31.840893 7269 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb1138d8852b41a5a19a537e0d7a94b0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cb1138d8852b41a5a19a537e0d7a94b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb1138d8852b41a5a19a537e0d7a94b0" member_type: VOTER last_known_addr { host: "127.16.121.190" port: 37733 } } peers { permanent_uuid: "fc986dfc94144d8ebf38873259309ba1" member_type: VOTER last_known_addr { host: "127.16.121.189" port: 43481 } } peers { permanent_uuid: "2a9a565db5ad4ba38312d27b9ab193b7" member_type: VOTER last_known_addr { host: "127.16.121.188" port: 40251 } } }
I20260928 22:53:31.841009 7269 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb1138d8852b41a5a19a537e0d7a94b0 [sys.catalog]: This master's current role is: LEADER
I20260928 22:53:31.842294 7271 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 22:53:31.843421 7271 catalog_manager.cc:1504] Loading token signing keys...
I20260928 22:53:31.850260 7271 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb1138d8852b41a5a19a537e0d7a94b0: Generated new TSK 0
I20260928 22:53:31.850525 7271 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 22:53:31.886966 7084 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:53:31.885486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41404 (local address 127.16.121.190:37733)
0928 22:53:31.885526 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:31.885528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:31.886269 (+ 741us) server_negotiation.cc:407] Connection header received
0928 22:53:31.886376 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:31.886377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:31.886389 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:31.886418 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:31.886892 (+ 474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41404: BlockingRecv error: recv got EOF from 127.0.0.1:41404 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 22:53:31.887956 7160 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:53:31.885276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34882 (local address 127.16.121.189:43481)
0928 22:53:31.885321 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:31.885323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:31.887427 (+ 2104us) server_negotiation.cc:407] Connection header received
0928 22:53:31.887526 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:31.887527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:31.887538 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:31.887568 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:31.887893 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34882: BlockingRecv error: recv got EOF from 127.0.0.1:34882 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 22:53:31.888813 7234 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:53:31.885969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41338 (local address 127.16.121.188:40251)
0928 22:53:31.886011 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:31.886013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:31.888300 (+ 2287us) server_negotiation.cc:407] Connection header received
0928 22:53:31.888396 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:31.888397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:31.888408 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:31.888436 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:31.888742 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41338: BlockingRecv error: recv got EOF from 127.0.0.1:41338 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 22:53:31.891301 7160 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:53:31.890469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34896 (local address 127.16.121.189:43481)
0928 22:53:31.890504 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:31.890506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:31.890713 (+ 207us) server_negotiation.cc:407] Connection header received
0928 22:53:31.890841 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:31.890842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:31.890861 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:31.890891 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:31.891255 (+ 364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34896: BlockingRecv error: recv got EOF from 127.0.0.1:34896 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 22:53:31.894450 7160 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:53:31.893676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34902 (local address 127.16.121.189:43481)
0928 22:53:31.893710 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:31.893712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:31.893943 (+ 231us) server_negotiation.cc:407] Connection header received
0928 22:53:31.894042 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:31.894043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:31.894054 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:31.894083 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:31.894403 (+ 320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34902: BlockingRecv error: recv got EOF from 127.0.0.1:34902 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 22:53:31.898447 7160 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:53:31.897708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34908 (local address 127.16.121.189:43481)
0928 22:53:31.897741 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:31.897743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:31.897975 (+ 232us) server_negotiation.cc:407] Connection header received
0928 22:53:31.898067 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:31.898068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:31.898077 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:31.898104 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:31.898400 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34908: BlockingRecv error: recv got EOF from 127.0.0.1:34908 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 22:53:31.904783 7160 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:53:31.903829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34922 (local address 127.16.121.189:43481)
0928 22:53:31.903869 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:31.903871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:31.904158 (+ 287us) server_negotiation.cc:407] Connection header received
0928 22:53:31.904279 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:31.904281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:31.904293 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:31.904325 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:31.904716 (+ 391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34922: BlockingRecv error: recv got EOF from 127.0.0.1:34922 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 22:53:31.915359 7160 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:53:31.914278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34932 (local address 127.16.121.189:43481)
0928 22:53:31.914321 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:31.914323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:31.914626 (+ 303us) server_negotiation.cc:407] Connection header received
0928 22:53:31.914740 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:31.914741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:31.914754 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:31.914800 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:31.915295 (+ 495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34932: BlockingRecv error: recv got EOF from 127.0.0.1:34932 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 22:53:31.933893 7160 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:53:31.932913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34948 (local address 127.16.121.189:43481)
0928 22:53:31.932957 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:31.932959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:31.933255 (+ 296us) server_negotiation.cc:407] Connection header received
0928 22:53:31.933363 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:31.933365 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:31.933377 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:31.933408 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:31.933828 (+ 420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34948: BlockingRecv error: recv got EOF from 127.0.0.1:34948 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 22:53:31.968290 7160 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:53:31.967359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34952 (local address 127.16.121.189:43481)
0928 22:53:31.967402 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:31.967404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:31.967717 (+ 313us) server_negotiation.cc:407] Connection header received
0928 22:53:31.967821 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:31.967822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:31.967834 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:31.967863 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:31.968230 (+ 367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34952: BlockingRecv error: recv got EOF from 127.0.0.1:34952 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 22:53:32.034662 7160 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:53:32.033981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34956 (local address 127.16.121.189:43481)
0928 22:53:32.034067 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:32.034069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:32.034110 (+ 41us) server_negotiation.cc:407] Connection header received
0928 22:53:32.034187 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:32.034188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:32.034201 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:32.034269 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:32.034612 (+ 343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34956: BlockingRecv error: recv got EOF from 127.0.0.1:34956 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260928 22:53:32.164932 7160 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:53:32.164126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34966 (local address 127.16.121.189:43481)
0928 22:53:32.164180 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:32.164183 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:32.164299 (+ 116us) server_negotiation.cc:407] Connection header received
0928 22:53:32.164412 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:32.164413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:32.164426 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:32.164490 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:32.164864 (+ 374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34966: BlockingRecv error: recv got EOF from 127.0.0.1:34966 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260928 22:53:32.423005 7160 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 22:53:32.422271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34972 (local address 127.16.121.189:43481)
0928 22:53:32.422361 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:32.422363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:32.422432 (+ 69us) server_negotiation.cc:407] Connection header received
0928 22:53:32.422521 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:32.422523 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:32.422536 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:32.422591 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 22:53:32.422956 (+ 365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34972: BlockingRecv error: recv got EOF from 127.0.0.1:34972 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 22:53:32.780293 7263 catalog_manager.cc:1259] Loaded cluster ID: aa8b0680d11844a7ac5adf3b8510e2ce
I20260928 22:53:32.780371 7263 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2a9a565db5ad4ba38312d27b9ab193b7: loading cluster ID for follower catalog manager: success
I20260928 22:53:32.781255 7263 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2a9a565db5ad4ba38312d27b9ab193b7: acquiring CA information for follower catalog manager: success
I20260928 22:53:32.781517 7263 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2a9a565db5ad4ba38312d27b9ab193b7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 22:53:32.836506 7292 catalog_manager.cc:1259] Loaded cluster ID: aa8b0680d11844a7ac5adf3b8510e2ce
I20260928 22:53:32.836596 7292 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fc986dfc94144d8ebf38873259309ba1: loading cluster ID for follower catalog manager: success
I20260928 22:53:32.837502 7292 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fc986dfc94144d8ebf38873259309ba1: acquiring CA information for follower catalog manager: success
I20260928 22:53:32.837858 7292 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fc986dfc94144d8ebf38873259309ba1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 22:53:32.938824 7365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:32.936112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34982 (local address 127.16.121.189:43481)
0928 22:53:32.936226 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:32.936228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:32.936270 (+ 42us) server_negotiation.cc:407] Connection header received
0928 22:53:32.936295 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:32.936296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:32.936345 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:32.936414 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:53:32.936614 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:32.937131 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:32.938043 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:32.938264 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:32.938321 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:32.938487 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:53:32.938599 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:53:32.938661 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:32.938732 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 22:53:32.938753 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":62,"threads_started":1}
I20260928 22:53:32.943482 7371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:32.940893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41354 (local address 127.16.121.188:40251)
0928 22:53:32.941011 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:32.941013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:32.941022 (+ 9us) server_negotiation.cc:407] Connection header received
0928 22:53:32.941071 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:32.941072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:32.941108 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:32.941170 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:53:32.941238 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:32.941723 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:32.942620 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:32.942809 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:32.942882 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:32.943078 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:53:32.943194 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:53:32.943255 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:32.943338 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 22:53:32.943364 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":55,"threads_started":1}
I20260928 22:53:32.948665 7371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:32.945704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41364 (local address 127.16.121.188:40251)
0928 22:53:32.945743 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:32.945745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:32.945962 (+ 217us) server_negotiation.cc:407] Connection header received
0928 22:53:32.946055 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:32.946056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:32.946066 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:32.946118 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:53:32.946311 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:32.947012 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:32.947823 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:32.948118 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:32.948178 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:32.948382 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:53:32.948415 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:53:32.948464 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:32.948564 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 22:53:32.948588 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 22:53:32.948702 7365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:32.945423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34984 (local address 127.16.121.189:43481)
0928 22:53:32.945547 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:32.945550 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:32.945716 (+ 166us) server_negotiation.cc:407] Connection header received
0928 22:53:32.945783 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:32.945784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:32.945795 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:32.945847 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:53:32.945957 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:32.946813 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:32.947886 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:32.948185 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:32.948244 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:32.948414 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:53:32.948445 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:53:32.948514 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:32.948585 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 22:53:32.948609 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260928 22:53:32.948833 7378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 22:53:32.945439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41414 (local address 127.16.121.190:37733)
0928 22:53:32.945603 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 22:53:32.945606 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 22:53:32.945614 (+ 8us) server_negotiation.cc:407] Connection header received
0928 22:53:32.945662 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 22:53:32.945663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 22:53:32.945717 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 22:53:32.945823 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 22:53:32.945977 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:32.946689 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:32.947865 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 22:53:32.948100 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 22:53:32.948183 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 22:53:32.948381 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 22:53:32.948501 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 22:53:32.948706 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0928 22:53:32.948724 (+ 18us) server_negotiation.cc:299] Negotiation successful
0928 22:53:32.948749 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":57,"threads_started":1}
I20260928 22:53:32.956355 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 7013
I20260928 22:53:32.962152 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 7088
I20260928 22:53:32.967288 16870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhFq4Cy/build/release/bin/kudu with pid 7163
2026-09-28T22:53:32Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2928 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5866 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (279644 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST